Image processing device, electronic apparatus, information storage device, and image processing method

ABSTRACT

The image processing device includes a first image acquisition section that acquires a first image that includes an object image including information within a wavelength band of white light, a second image acquisition section that acquires a second image that includes an object image including information within a specific wavelength band, a type determination section that determines a type of the object image in the second image based on a feature quantity of each pixel included in the second image, and a highlight section that performs a highlight process on the first image based on the type of the object image determined by the type determination section.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of International Patent ApplicationNo. PCT/JP2010/071963, having an international filing date of Dec. 8,2010, which designated the United States, the entirety of which isincorporated herein by reference. Japanese Patent Application No.2009-296964 filed on Dec. 28, 2009 is also incorporated herein byreference in its entirety.

BACKGROUND

The present invention relates to an image processing device, anelectronic apparatus, an information storage device, an image processingmethod, and the like.

A frame-sequential endoscope system has been widely used. Theframe-sequential endoscope system sequentially applies three colors oflight (R1, G1, and B1) to tissue in a body cavity using a rotary filter,and allows the user to perform diagnosis using an image (normal lightimage) generated from reflected light images. An endoscope system hasbeen proposed that sequentially applies two types of narrow-band light(G2 and B2) that differs in characteristics from the three colors oflight to tissue in a body cavity, and allows the user to performdiagnosis using a narrow-band light image generated from reflected lightimages (see JP-A-2006-68113, for example). An endoscope system has alsobeen proposed that applies narrow-band excitation light to tissue in abody cavity, and allows the user to perform diagnosis using afluorescent image generated by acquiring intrinsic fluorescence producedby the tissue or fluorescence produced by a fluorescent agent due to theexcitation light (see JP-A-2007-229053, for example).

When performing diagnosis using an endoscope system that acquires anarrow-band light image (e.g., JP-A-2006-68113), a lesion area (e.g.,epidermoid cancer) that is difficult to observe in a normal light imageis visualized as a brown area differing from a normal area. Therefore, alesion area can be easily found using such an endoscope system.

When performing diagnosis using an endoscope system that acquires afluorescent image (e.g., JP-A-2007-229053), only a lesion area (e.g.,tumor) produces fluorescence by utilizing a fluorescent agent that isspecifically accumulated in such a lesion area. Therefore, a lesion areacan be easily found using such an endoscope system.

SUMMARY

According to one aspect of the invention, there is provided an imageprocessing device comprising:

a first image acquisition section that acquires a first image thatincludes an object image including information within a wavelength bandof white light;

a second image acquisition section that acquires a second image thatincludes an object image including information within a specificwavelength band;

a type determination section that determines a type of the object imagein the second image based on a feature quantity of each pixel includedin the second image; and

a highlight section that performs a highlight process on the first imagebased on the type of the object image determined by the typedetermination section.

According to another aspect of the invention, there is provided anelectronic apparatus comprising:

the image processing device.

According to another aspect of the invention, there is provided an imageprocessing device comprising:

a first image acquisition section that acquires a first image thatincludes an object image including information within a wavelength bandof white light;

a second image acquisition section that acquires a second image thatincludes an object image including information within a specificwavelength band; and

a highlight section that performs a highlight process on the first imagebased on a type of the second image acquired by the second imageacquisition section.

According to another aspect of the invention, there is provided aninformation storage device storing a program that causes a computer tofunction as:

a first image acquisition section that acquires a first image thatincludes an object image including information within a wavelength bandof white light;

a second image acquisition section that acquires a second image thatincludes an object image including information within a specificwavelength band;

a type determination section that determines a type of the object imagein the second image based on a feature quantity of each pixel includedin the second image; and

a highlight section that performs a highlight process on the first imagebased on the type of the object image determined by the typedetermination section.

According to another aspect of the invention, there is provided aninformation storage device storing a program that causes a computer tofunction as:

a first image acquisition section that acquires a first image thatincludes an object image including information within a wavelength bandof white light;

a second image acquisition section that acquires a second image thatincludes an object image including information within a specificwavelength band; and

a highlight section that performs a highlight process on the first imagebased on a type of the second image acquired by the second imageacquisition section.

According to another aspect of the invention, there is provided an imageprocessing method comprising:

acquiring a first image that includes an object image includinginformation within a wavelength band of white light;

acquiring a second image that includes an object image includinginformation within a specific wavelength band;

determining a type of the object image in the second image based on afeature quantity of each pixel included in the second image; and

performing a highlight process on the first image based on thedetermined type of the object image.

According to another aspect of the invention, there is provided an imageprocessing method comprising:

acquiring a first image that includes an object image includinginformation within a wavelength band of white light;

acquiring a second image that includes an object image includinginformation within a specific wavelength band; and

performing a highlight process on the first image based on a type of theacquired second image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view illustrating a type determination process and ahighlight process.

FIG. 2 illustrates a system configuration example according to oneembodiment of the invention.

FIG. 3 illustrates the spectral characteristics of color filters r, g,and b.

FIG. 4 is a view illustrating color filters g2 and b2.

FIG. 5 illustrates the spectral characteristics of color filters g2 andb2.

FIG. 6 illustrates a configuration example of a normal light imageacquisition section.

FIG. 7 illustrates a configuration example of a special light imageacquisition section.

FIG. 8 is a view illustrating color filters g2 and b2.

FIG. 9 illustrates a configuration example of an area type determinationsection.

FIG. 10 is a view illustrating a local area setting method.

FIG. 11 illustrates a configuration example of a computer used for asoftware process.

FIG. 12 illustrates a configuration example of a computer used for asoftware process.

FIG. 13 is a flowchart illustrating a process according to oneembodiment of the invention.

FIG. 14 is a flowchart illustrating an area type determination process.

FIG. 15 is another view illustrating a type determination process and ahighlight process.

FIG. 16 illustrates another configuration example of an area typedetermination section.

FIG. 17 illustrates a configuration example of a highlight section.

FIG. 18 illustrates a configuration example of a high-frequencycomponent calculation section.

FIG. 19 is a view illustrating a multiple resolution-transformed imageobtained by a wavelet transform.

FIG. 20 is a view illustrating an endoscopic image.

FIG. 21 is a view illustrating the relationship between a blood vesseland a mucous membrane in a multiple resolution-transformed image.

FIG. 22 is another flowchart, illustrating an area type determinationprocess.

FIG. 23 is a flowchart illustrating a highlight process.

FIG. 24 is another view illustrating a type determination process and ahighlight process.

FIG. 25 illustrates another configuration example of an area typedetermination section.

FIG. 26 illustrates another configuration example of a highlightsection.

FIG. 27 is another flowchart illustrating an area type determinationprocess.

FIG. 28 illustrates another system configuration example according toone embodiment of the invention.

FIG. 29 illustrates another configuration example of a special lightimage acquisition section.

FIG. 30 is another view illustrating a type determination process and ahighlight process.

FIG. 31 illustrates another system configuration example according toone embodiment of the invention.

FIG. 32 illustrates a configuration example of a rotary filter.

FIG. 33 illustrates the spectral characteristics of a filter F1.

FIG. 34 illustrates the spectral characteristics of a filter F2.

FIG. 35 is a view illustrating color filters g3 and r3.

FIG. 36 illustrates the spectral characteristics of color filters g3 andr3.

FIG. 37 is a view illustrating color filters g3 and r3.

FIG. 38 illustrates an example of the combination of a filter and theresulting image at each timing.

FIG. 39 illustrates an example of acquisition timings of a normal lightimage and a special light image.

FIG. 40 is another flowchart illustrating a process according to oneembodiment of the invention.

FIG. 41 is another flowchart illustrating an area type determinationprocess.

FIG. 42 is another view illustrating a type determination process and ahighlight process.

FIG. 43 illustrates yet another system configuration example accordingto one embodiment of the invention.

FIG. 44 illustrates a configuration example of a rotary filter.

FIG. 45 illustrates a configuration example of a rotary filter.

FIG. 46 illustrates the spectral characteristics of a filter F3.

FIG. 47 illustrates the spectral characteristics of a filter F4.

FIG. 48 is a view illustrating color filters g4 and b4.

FIG. 49 illustrates the spectral characteristics of a color filter g4.

FIG. 50 illustrates the spectral characteristics of a color filter b4.

FIG. 51 illustrates another configuration example of a highlightsection.

FIG. 52 is another flowchart illustrating a process according to oneembodiment of the invention.

FIG. 53 is another flowchart illustrating a highlight process.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

According to one embodiment of the invention, there is provided an imageprocessing device comprising:

a first image acquisition section that acquires a first image thatincludes an object image including information within a wavelength bandof white light;

a second image acquisition section that acquires a second image thatincludes an object image including information within a specificwavelength band;

a type determination section that determines a type of the object imagein the second image based on a feature quantity of each pixel includedin the second image; and

a highlight section that performs a highlight process on the first imagebased on the type of the object image determined by the typedetermination section.

According to above embodiment of the invention, the first image thatcorresponds to the wavelength band of the white light and the secondimage that corresponds to the specific wavelength band are acquired. Thetype of the object image in the second image is determined, and thehighlight process is performed on the first image based on the type ofthe object image. This makes it possible to perform various highlightprocesses corresponding to the situation.

According to another embodiment of the invention, there is provided animage processing device comprising:

a first image acquisition section that acquires a first image thatincludes an object image including information within a wavelength bandof white light;

a second image acquisition section that acquires a second image thatincludes an object image including information within a specificwavelength band; and

a highlight section that performs a highlight process on the first imagebased on a type of the second image acquired by the second imageacquisition section.

According to above embodiment of the invention, the first image thatcorresponds to the wavelength band of the white light and the secondimage that corresponds to the specific wavelength band are acquired, andthe highlight process is performed on the first image based on thesecond image. This makes it possible to perform various highlightprocesses corresponding to the situation even if the second image may beclassified into a plurality of types.

According to another embodiment of the invention, there is provided anelectronic apparatus comprising:

the image processing device.

According to another embodiment of the invention, there is provided aninformation storage device storing a program that causes a computer tofunction as above sections.

According to another embodiment of the invention, there is provided animage processing method comprising:

acquiring a first image that includes an object image includinginformation within a wavelength band of white light;

acquiring a second image that includes an object image includinginformation within a specific wavelength band;

determining a type of the object image in the second image based on afeature quantity of each pixel included in the second image; and

performing a highlight process on the first image based on thedetermined type of the object image.

According to another embodiment of the invention, there is provided animage processing method comprising:

acquiring a first image that includes an object image includinginformation within a wavelength band of white light;

acquiring a second image that includes an object image includinginformation within a specific wavelength band; and

performing a highlight process on the first image based on a type of theacquired second image.

Exemplary embodiments of the invention are described below. Note thatthe following exemplary embodiments do not in any way limit the scope ofthe invention laid out in the claims. Note also that all of the elementsof the following exemplary embodiments should not necessarily be takenas essential elements of the invention.

1. First Embodiment

An outline of a first embodiment of the invention is described belowwith reference to FIG. 1.

Several aspects and embodiments of the invention propose a method thatchanges a highlight process performed on a first image depending on thetype of object image within a second image. In the first embodiment, asecond image is an NBI (narrow band imaging) image (see A1). The type ofan object image in the second image is determined (A2). In the exampleillustrated in FIG. 1, the hue H is used as a feature quantity fordetermining the type of the object image. An object image having a hue Hof 5 to 35 is determined to be a first type of object image, and anobject image having a hue H of 170 to 200 is determined to be a secondtype of object image (see A3). An object image other than the first typeof object image and the second type of object image is determined to bea third type of object image. The first type of object image correspondsto a blood vessel in a surface area, and the second type of object imagecorresponds to a blood vessel in a deep area. The third type of objectimage corresponds to an area (e.g., mucous membrane) other than a bloodvessel in a surface area and a blood vessel in a deep area.

A highlight range is determined after determining the type of the objectimage. In the first embodiment, the highlight range corresponds to thefirst type and the second type (i.e., a blood vessel in a surface areaand a blood vessel in a deep area) (A4). Therefore, the first image(normal light image in a narrow sense) is highlighted as indicated byA5.

A highlight process is performed on the highlight range using a givenmethod. In the first embodiment, the highlight process is implementedusing a color conversion process (see A6). The color conversion processis performed using expressions (9) and (10). The highlight processcorresponding to the type of the object image can be performed bychanging the target color and the gain parameter depending on the type(first type or second type) of object image. A normal light image inwhich the blood vessel is highlighted (see A7) can thus be acquired.Since the first type of object image and the second type of object imagediffer in target color, a blood vessel in a surface area and a bloodvessel in a deep area are displayed in a different color.

The method of performing the highlight process may be changedcorresponding to the organ to be observed.

An endoscope system according to the first embodiment is described belowwith reference to FIG. 2. The endoscope system according to the firstembodiment includes a light source section 100, an imaging section 200,an image processing section 300, a display section 400, and an externalI/F section 500.

The light source section 100 includes a white light source 110 thatemits white light, and a condenser lens 120 that focuses the white lightemitted from the white light source 110 on a light guide fiber 210.

The imaging section 200 is formed to be elongated and flexible (i.e.,can be curved) so that the imaging section 200 can be inserted into abody cavity or the like.

The imaging section 200 can be removed so that a different imagingsection can be used depending on the organ to be observed. The imagingsection 200 is normally referred to as “scope” in the field ofendoscopy. Specific examples of the scope include an uppergastrointestinal scope, a lower gastrointestinal scope, and the like.

The imaging section 200 includes the light guide fiber 210 that guideslight focused by the light source section 100, an illumination lens 220that diffuses light that has been guided by the light guide fiber 210,and illuminates an object, an objective lens 230 that focuses lightreflected by the object, a half mirror 240 that divides (separates) thefocused reflected light into two parts, and a first imaging element 250and a second imaging element 260 that detect the reflected light dividedby the half mirror 240. The first imaging element 250 includes a Bayercolor filter array that is used to capture a normal light image. Forexample, color filters R, G, and B of the first imaging element 250 havethe spectral characteristics illustrated in FIG. 3. As illustrated inFIG. 4, the second imaging element 260 has a configuration in whichcolor filters g2 and b2 are disposed in a staggered arrangement, forexample. As illustrated in FIG. 5, the color filter b2 allows lightwithin a wavelength band of 390 to 445 nm to pass through, and the colorfilter g2 allows light within a wavelength band of 530 to 550 nm to passthrough, for example.

The imaging section 200 includes a memory 280. The memory 280 stores anidentification number specific to each scope. The memory 280 isconnected to a control section 360. The control section 360 can specifythe scope referring to the identification number stored in the memory280. The control section 360 can also specify the organ to be observedby specifying the scope.

The image processing section 300 includes AD conversion sections 310 and311, a normal light image acquisition section 320, a special light imageacquisition section 330, an area type determination section 340, ahighlight section 350, and the control section 360. The control section360 is connected to the normal light image acquisition section 320, thespecial light image acquisition section 330, the area type determinationsection 340, and the highlight section 350, and controls the normallight image acquisition section 320, the special light image acquisitionsection 330, the area type determination section 340, and the highlightsection 350.

The external I/F section 500 is an interface that allows the user toperform an input operation or the like on the image processing device.The external I/F section 500 includes a power switch (power ON/OFFswitch), a shutter button (capture operation start button), a mode(e.g., imaging mode) change button, and the like. The external I/Fsection 500 outputs input information to the control section 360.

The AD conversion section 310 converts an analog image signal outputfrom the first imaging element 250 into a digital image signal, andoutputs the digital image signal. The AD conversion section 311 convertsan analog image signal output from the second imaging element 260 into adigital image signal, and outputs the digital image signal.

The normal light image acquisition section 320 acquires a normal lightimage from the digital image signal output from the AD conversionsection 310. The special light image acquisition section 330 acquires aspecial light image from the digital image signal output from the A/Dconversion section 311. The details of the normal light imageacquisition section 320 and the special light image acquisition section330 are described later.

The normal light image acquired by the normal light image acquisitionsection 320 is output to the highlight section 350. The special lightimage acquired by the special light image acquisition section 330 isoutput to the area type determination section 340. The area typedetermination section 340 determines the type of the object in thespecial light image, and outputs the determination result to thehighlight section 350. The highlight section 350 performs a highlightprocess on the normal light image corresponding to the determinationresult output from the area type determination section 340, and outputsthe resulting normal light image to the display section 400. The detailsof the area type determination section 340 and the highlight section 350are described later.

The details of the normal light image acquisition section 320 aredescribed below with reference to FIG. 6. The normal light imageacquisition section 320 includes a normal light image generation section321 and a normal light image storage section 322. The normal light imagegeneration section 321 performs image processing on the digital imagesignal output from the AD conversion section 310 to generate a normallight image. More specifically, the normal light image generationsection 321 performs an interpolation process, a white balance process,a color conversion process, a grayscale transformation process, and thelike on the digital image signal to generate a normal light image, andoutputs the normal light image. The normal light image is an RGB image.The normal light image storage section 322 stores the normal light imageoutput from the normal light image generation section 321 in a memory.

The details of the special light image acquisition section 330 aredescribed below with reference to FIG. 7. The special light imageacquisition section 330 includes a special light image generationsection 331 and a special light image storage section 332. The speciallight image generation section 331 performs image processing on thedigital image signal output from the A/D conversion section 311 togenerate a special light image. In the first embodiment, the speciallight image is a narrow-band light image. The narrow-band light image isan RGB image.

The special light image generation section 331 generates the narrow-bandlight image as described below. The second imaging element 260 has aconfiguration in which the color filters g2 and b2 are disposed in astaggered arrangement (see FIG. 4). Therefore, digital image signalsillustrated in FIG. 8 are input to the special light image generationsection 331. Note that G2(x, y) indicates the signal value of the g2filter, and B2(x, y) indicates the signal value of the b2 filter. x andy are image coordinate values. The special light image generationsection 331 performs an interpolation process on the image signals togenerate a G2 image in which each pixel has the signal value of thefilter g2, and a B2 image in which each pixel has the signal value ofthe filter b2. The signal value calculated by the interpolation processmay be the average value of the signal values of four peripheral pixels.For example, the signal value B2(1, 1) of the filter b2 at the positionG2(1, 1) and the pixel value G2(1, 2) of the filter g2 at the positionB2(1, 2) illustrated in FIG. 8 are calculated by the followingexpressions (1) and (2).B2(1,1)=[B2(0, 1)+B2(1, 0)+B2(1, 2)+B2(2, 1)]/4  (1)G2(1, 2)=[G2(0, 2)+G2(1, 1)+G2(1, 3)+G2(2, 2)]/4  (2)

A color image having R, G, and B signal values is generated from the G2image and the B2 image generated by the interpolation process. The colorimage is generated by inputting the signal value G2(x, y) to the Rsignal at the coordinates (x, y), and inputting the signal value B2(x,y) to the G and B signals at the coordinates (x, y). The special lightimage generation section 331 performs a white balance process, agrayscale transformation process, and the like on the generated colorimage to generate a narrow-band light image. The narrow-band light imagethus generated is output to the special light image storage section 332.The special light image storage section 332 stores the narrow-band lightimage output from the special light image generation section 331 in amemory.

The details of the area type determination section 340 are describedbelow. FIG. 9 is a block diagram illustrating an example of theconfiguration of the area type determination section 340 according tothe first embodiment. The area type determination section 340 includes alocal area setting section 341, a feature quantity calculation section342, a type determination section 343, and an area selection section344. The control section 360 is connected to the local area settingsection 341, the feature quantity calculation section 342, the typedetermination section 343, and the area selection section 344, andcontrols the local area setting section 341, the feature quantitycalculation section 342, the type determination section 343, and thearea selection section 344.

The local area setting section 341 sets a plurality of local areaswithin the narrow-band light image output from the special light imageacquisition section 330. For example, the local area setting section 341divides the narrow-band light image into a plurality of rectangularareas, and sets the plurality of rectangular areas as the local areas.For example, the local area setting section 341 sets a 5×5 pixel area asone local area (see FIG. 10). The narrow-band light image includes M×Nlocal areas, and the coordinates of each local area are indicated by (m,n). The local area positioned at the coordinates (m, n) is indicated bya(m, n). The coordinates of the local area positioned at the upper leftof the image are indicated by (0, 0). The rightward direction is thepositive direction of the coordinate value m, and the downward directionis the positive direction of the coordinate value n. Note that the localarea need not necessarily be rectangular. The narrow-band light imagemay be divided into a plurality of arbitrary polygonal areas, and theplurality of polygonal areas may be set as the local areas. The localareas may be arbitrarily set based on instructions from the user. In thefirst embodiment, each local area includes a plurality of adjacentpixels in order to reduce the amount of calculations. Note that onepixel may be set as one local area. In this case, the subsequent processis performed in the same manner as in the case where each local areaincludes a plurality of adjacent pixels.

The feature quantity calculation section 342 calculates the featurequantity of each local area set by the local area setting section 341. Acase where hue is used as the feature quantity is described below.

The hue of the local area a(m, n) is indicated by H(m, n). Whencalculating the hue H(m, n), the average values R_ave, G_ave, and B_aveof the R, G, and B signals in each local area are calculated. Theaverage value R_ave is the average value of the R signal of each pixelincluded in each local area. The average value G_ave is the averagevalue of the G signal of each pixel included in each local area, and theaverage value B_ave is the average value of the B signal of each pixelincluded in each local area. The signal value is indicated by 8 bits (0to 255). The hue H(m, n) of each local area is calculated by thefollowing expressions (3) to (8) using the average values R_ave, G_ave,and B_ave, for example.Max=MAX(R_ave, G_ave, B_ave)  (3)

The MAX function outputs the maximum argument among a plurality ofarguments.

When max is 0:H=0  (4)

When max is not 0:d=MAX(R_ave, G_ave, B_ave)−MIN(R_ave, G_ave, B_ave)  (5)

The MIN function outputs the minimum argument among a plurality ofarguments.

When the average value R_ave is a maximum among the average valuesR_ave, G_ave, and B_ave:H=60*(G_ave−B_ave)/d  (6)

When the average value G_ave is a maximum among the average valuesR_ave, G_ave, and B_ave:H=60*{2+(B_ave−R_ave)}/d  (7)

When the average value B_ave is a maximum among the average valuesR_ave, G_ave, and B_ave:H=60*{4+(R_ave−G_ave)}/d  (8)

When the hue H is smaller than 0, 360 is added to the hue H. The hue His considered to be 0 when the hue H is 360.

The type determination section 343 determines the type of the object ineach local area using the hue H calculated for each local area, andoutputs the determination result to the area selection section 344.

When light emitted from the light source section 100 is applied totissue (i.e., object), short-wavelength light is reflected near thesurface of the tissue, while long-wavelength light reaches a deep areaof the tissue. The color filter b2 used for the imaging element thatcaptures the narrow-band light image allows light within a wavelengthband of 390 to 445 nm to pass through, and the color filter g2 used forthe imaging element that captures the narrow-band light image allowslight within a wavelength band of 530 to 550 nm to pass through.Therefore, short-wavelength light reflected by a surface area (layer) ofthe tissue passes through the b2 filter, and long-wavelength lightreflected by a deep area (layer) of the tissue passes through the g2filter. Light within the passband of each color filter is easilyabsorbed by hemoglobin that is abundantly contained in blood. Therefore,a blood vessel in the surface area of the tissue is drawn in the B2image, and a blood vessel in the deep area of the tissue is drawn in theG2 image.

When generating a color image from the B2 image and the G2 image, thesignal value G2(x, y) is input to the R signal at the coordinates (x,y), and the signal value B2(x, y) is input to the G and B signals at thecoordinates (x, y). Therefore, the R signal of the narrow-band lightimage includes information about a blood vessel in the deep area of thetissue, and the G and B signals of the narrow-band light image includeinformation about a blood vessel in the surface area of the tissue.

Therefore, a blood vessel in the surface area of the tissue is drawn inthe narrow-band light image as a brown area, and a blood vessel in thedeep area of the tissue is drawn in the narrow-band light image as ablue-green area. Specifically, since a blood vessel in the surface areaof the tissue and a blood vessel in the deep area of the tissue aredrawn to differ in hue, a blood vessel in the surface area of the tissueand a blood vessel in the deep area of the tissue can be discriminatedby utilizing the hue H as the feature quantity. For example, an areahaving a hue H of 5 to 35 may be determined to be a blood vessel in thesurface area of the tissue, and an area having a hue H of 170 to 200 maybe determined to be a blood vessel in the deep area of the tissue.

The type determination section 343 outputs the coordinates of each localarea that has been set by the local area setting section 341 anddetermined to be a blood vessel in the surface area of the tissue or ablood vessel in the deep area of the tissue, and tag information thatindicates the local area type determination result to the area selectionsection 344. The tag value may be set to “1” when the local area hasbeen determined to be a blood vessel in the surface area of the tissue,and may be set to “2” when the local area has been determined to be ablood vessel in the deep area of the tissue, for example.

The area selection section 344 calculates the position of each pixelincluded in each local area from the coordinates of the local area a(m,n) that has been determined to be a blood vessel in the surface area ofthe tissue or a blood vessel in the deep area of the tissue by the typedetermination section 343, and information about each pixel included ineach local area, adds the tag information to the calculated pixelposition information, and outputs the pixel position information to thehighlight section 350.

The details of the highlight section 350 are described below. Thehighlight section 350 performs the highlight process on each pixel ofthe normal light image that corresponds to the pixel position outputfrom the area type determination section 340. The highlight process on apixel (tag value: 1) that has been determined to be a blood vessel inthe surface area of the tissue may be implemented by a color conversionprocess shown by the following expression (9), and the highlight processon a pixel (tag value: 2) that has been determined to be a blood vesselin the deep area of the tissue may be implemented by a color conversionprocess shown by the following expression (10), for example.R_out(x, y)=gain*R(x, y)+(1−gain)*T_R1G_out(x, y)=gain*G(x, y)+(1−gain)*T _(—) G1B_out(x, y)=gain*B(x, y)+(1−gain)*T _(—) B1  (9)R_out(x, y)=gain*R(x, y)+(1−gain)*T _(—) R2G_out(x, y)=gain*G(x, y)+(1−gain)*T _(—) G2B_out(x, y)=gain*B(x, y)+(1−gain)*T _(—) B2  (10)where, R(x, y), G(x, y), and B(x, y) are the RGB signal values at thecoordinates (x, y) of the normal light image before the color conversionprocess, R_out(x, y), G_out(x, y), and B_out(x, y) are the RGB signalvalues at the coordinates (x, y) of the normal light image after thecolor conversion process, T_R1, T_G1, and T_B1 are the RGB signal valuesof the target color of a blood vessel in the surface area of the tissue,and T_R2, T_G2, and T_B2 are the RGB signal values of the target colorof a blood vessel in the deep area of the tissue. A blood vessel in thesurface area of the tissue and a blood vessel in the deep area of thetissue can be drawn in a different color by setting a different value asthe target color. gain is an arbitrary coefficient in the range of 0 to1.

Note that the target colors T_R1, T_G1, and T_B1 of a blood vessel inthe surface area of the tissue, the target colors T_R2, T_G2, and T_B2of a blood vessel in the deep area of the tissue, and the gain parametermay be set by the user via the external I/F section 500, or differentparameters may be set in advance corresponding to the organ to beobserved. The control section 360 may specify the organ to be observedreferring to the identification number specific to each scope stored inthe memory 280, or the user may designate the organ to be observed viathe external I/F section 500.

Since a blood vessel in the surface area of the tissue and a bloodvessel in the deep area of the tissue are drawn in a different color byperforming the above process, a blood vessel in the surface area of thetissue and a blood vessel in the deep area of the tissue can be easilydiscriminated. This makes it possible to prevent a situation in which alesion area is missed while reducing the burden on the doctor duringdiagnosis using the normal light image and the narrow-band light image.

Although an example in which the highlight process is independentlyperformed on an area determined to be a blood vessel in the surface areaof the tissue and an area determined to be a blood vessel in the deeparea of the tissue has been described above, the invention is notlimited thereto. For example, only a blood vessel in the surface area ofthe tissue or a blood vessel in the deep area of the tissue may behighlighted.

The above configuration may be implemented by assigning a priority tothe type of object (e.g., a blood vessel in the surface area of thetissue and a blood vessel in the deep area of the tissue) in advance,and determining the method of performing the highlight processcorresponding to the type of object with the highest priority when thearea type determination section 340 has detected a plurality of types ofobject.

For example, when a tumor has been found during diagnosis of the largeintestine, information about the blood vessel structure on the surfaceof the tumor is important for determining whether the tumor is benign ormalignant. Therefore, a high priority is assigned to a blood vessel inthe surface area of the tissue during diagnosis of the large intestine.This makes it possible to perform the highlight process on only an areadetermined to be a blood vessel in the surface area of the tissue evenwhen the area type determination section 340 has detected a plurality oftypes of object. This improves the visibility of a blood vessel in thesurface area of the tissue, and prevents a situation in which a lesionarea is missed.

A disease such as an esophageal varix may be found during diagnosis ofthe gullet. Since an esophageal varix is present in a relatively deeparea of the tissue, information about a blood vessel in the deep area ofthe tissue is important for diagnosis. Therefore, a high priority isassigned to a blood vessel in the deep area of the tissue duringdiagnosis of the gullet. This makes it possible to perform the highlightprocess on only an area determined to be a blood vessel in the deep areaof the tissue. This improves the visibility of a blood vessel in thedeep area of the tissue, and is effective for preventing a situation inwhich an esophageal varix is missed.

The priority may be set by the user via the external I/F section 500, ormay be set in advance corresponding to the organ to be observed. Thecontrol section 360 may specify the organ to be observed referring tothe identification number specific to each scope stored in the memory280, or the user may designate the organ to be observed via the externalI/F section 500.

The highlight process performed by the highlight section 350 may beimplemented using an arbitrary luminance conversion process or anarbitrary color conversion process instead of the above color conversionprocess.

Although an example in which each section included in the imageprocessing section 300 is implemented by hardware has been describedabove, the configuration is not limited thereto. For example, a CPU mayperform the process of each section on an image acquired in advanceusing an imaging element such as a capsule endoscope. Specifically, theprocess of each section may be implemented by software by causing theCPU to execute a program. Alternatively, the process of each section maypartially be implemented by software.

When separately providing an imaging section, and implementing theprocess of each section included in the image processing section 300 bysoftware, a known computer system (e.g., work station or personalcomputer) may be used as the image processing device. A program (imageprocessing program) that implements the process of each section includedin the image processing section 300 may be provided in advance, andexecuted by the CPU of the computer system.

FIG. 11 is a system configuration diagram illustrating the configurationof a computer system 600 according to a modification. FIG. 12 is a blockdiagram illustrating the configuration of a main body 610 of thecomputer system 600. As illustrated in FIG. 11, the computer system 600includes the main body 610, a display 620 that displays information(e.g., image) on a display screen 621 based on instructions from themain body 610, a keyboard 630 that allows the user to input informationto the computer system 600, and a mouse 640 that allows the user todesignate an arbitrary position on the display screen 621 of the display620.

As illustrated in FIG. 12, the main body 610 of the computer system 600includes a CPU 611, a RAM 612, a ROM 613, a hard disk drive (HDD) 614, aCD-ROM drive 615 that receives a CD-ROM 660, a USB port 616 to which aUSB memory 670 is removably connected, an I/O interface 617 thatconnects the display 620, the keyboard 630, and the mouse 640, and a LANinterface 618 that is used to connect to a local area network or a widearea network (LAN/WAN) N1.

The computer system 600 is connected to a modem 650 that is used toconnect to a public line N3 (e.g., Internet). The computer system 600 isalso connected to personal computer (PC) 681 (i.e., another computersystem), a server 682, a printer 683, and the like via the LAN interface618 and the local area network or the large area network N1.

The computer system 600 implements the functions of the image processingdevice by reading an image processing program (e.g., an image processingprogram that implements a process described below with reference toFIGS. 13 and 14) recorded on a given recording medium, and executing theimage processing program. The given recording medium may be an arbitraryrecording medium that records the image processing program that can beread by the computer system 600, such as the CD-ROM 660, the USB memory670, a portable physical medium (e.g., MO disk, DVD disk, flexible disk(FD), magnetooptical disk, or IC card), a stationary physical medium(e.g., HDD 614, RAM 612, or ROM 613) that is provided inside or outsidethe computer system 600, or a communication medium that temporarilystores a program during transmission (e.g., the public line N3 connectedvia the modem 650, or the local area network or the wide area network N1to which the computer system (PC) 681 or the server 682 is connected).

Specifically, the image processing program is recorded on a recordingmedium (e.g., portable physical medium, stationary physical medium, orcommunication medium) so that the image processing program can be readby a computer. The computer system 600 implements the functions of theimage processing device by reading the image processing program fromsuch a recording medium, and executing the image processing program.Note that the image processing program need not necessarily be executedby the computer system 600. The invention may similarly be applied tothe case where the computer system (PC) 681 or the server 682 executesthe image processing program, or the computer system (PC) 681 and theserver 682 execute the image processing program in cooperation.

A process performed on the normal light image and the narrow-band lightimage acquired in advance when implementing the process of the area typedetermination section 340 and the process of the highlight section 350illustrated in FIG. 2 by software is described below using a flowchartillustrated in FIG. 13.

Specifically, the narrow-band light image is written into the memory(Step 1), and the normal light image acquired at the same time with thenarrow-band light image is then written into the memory (Step 2). Thetype of the object is determined using the narrow-band light image (Step3). The details of the area type determination step (Step 3 ) aredescribed later. The highlight process is performed on the normal lightimage corresponding to the determination result, and the image obtainedby the highlight process is output as a display image (Step 4). Thehighlight process is performed using the expressions (9) and (10). Theprocess ends when all of the images have been processed. The process isrepeated until all of the images have been processed (Step 5).

A specific process of the area type determination step (Step 3)illustrated in FIG. 13 is described below using a flowchart illustratedin FIG. 14. In a local area setting step, a plurality of local areas areset within the narrow-band light image (see FIG. 10) (Step 31). Thefeature quantity of each local area is calculated (Step 32). Forexample, the hue H indicated by the expressions (3) to (8) is used asthe feature quantity. A type determination process is performed based onthe hue H calculated for each local area (Step 33). More specifically,an area having a hue H of 5 to 35 is determined to be a blood vessel inthe surface area of the tissue, and an area having a hue H of 170 to 200is determined to be a blood vessel in the deep area of the tissue. Theposition of each pixel included in each local area is calculated fromthe coordinates of the local area a(m, n) that has been determined to bea blood vessel in the surface area of the tissue or a blood vessel inthe deep area of the tissue, and information about each pixel includedin each local area. The tag information that indicates the determinationresult is added to the calculated pixel position information, and theresulting pixel position information is output (Step 34).

Since a blood vessel in the surface area of the tissue and a bloodvessel in the deep area of the tissue are displayed in a different colorby performing the above process, a blood vessel in the surface area ofthe tissue and a blood vessel in the deep area of the tissue can beeasily discriminated. This makes it possible to prevent a situation inwhich a lesion area is missed while reducing the burden on the doctorduring diagnosis using the normal light image and the narrow-band lightimage.

According to the first embodiment, a first image acquisition section(normal light image acquisition section 320 in a narrow sense) acquiresa first image (white light image in a narrow sense) that corresponds tothe wavelength band of white light, and a second image acquisitionsection (special light image acquisition section 330 in a narrow sense)acquires a second image (special light image (e.g., narrow-band image orfluorescent image) in a narrow sense) that corresponds to a specificwavelength band (the wavelength band of narrow-band light orfluorescence in a narrow sense). The type determination section 343determines the type of an object image in the second image using thefeature quantity of each pixel included in the second image. Thehighlight section 350 performs the highlight process on the first imagebased on the determination result of the type determination section 343.

The term “type of object image” used herein refers to the type of lesionor the type of blood vessel. For example, the first type of object imageis a lesion area, and the second type of object image is a normal area.The first type of object image may be a blood vessel in a surface area,the second type of object image may be a blood vessel in a deep area,and the third type of object image may be an area (e.g., mucousmembrane) other than a blood vessel.

The feature quantity of each pixel may be the hue H, edge quantity E,signal value (R, G, and B), or the like. The feature quantity isappropriately selected corresponding to the object (i.e., a targetsubjected to the highlight process).

The above configuration makes it possible to acquire the normal lightimage (white light image) and the special light image (e.g., NBI imageor fluorescent image), and determine the type of the object image in thespecial light image. The method of performing the highlight processperformed on the normal light image can be changed based on thedetermined type of the object image. Therefore, it is possible todiscriminate a lesion area and a normal area, and perform the highlightprocess on only the lesion area, for example. It is also possible todiscriminate a blood vessel in a surface area and a blood vessel in adeep area, and change the method of performing the highlight processbetween the blood vessel in the surface area and the blood vessel in thedeep area, for example.

The highlight section 350 may change the method of performing thehighlight process corresponding to an organ to which the object imagebelongs.

This makes it possible to change the method of performing the highlightprocess depending on the organ to be observed. For example, when thefirst type of object image is a blood vessel in a surface area, thesecond type of object image is a blood vessel in a deep area, and thethird type of object image is an area (e.g., mucous membrane) other thana blood vessel, information about a blood vessel in a surface area of atumor is important for determining whether the tumor is benign ormalignant when observing the large intestine. In this case, a bloodvessel in a surface area (i.e., first type of object image) may bepreferentially highlighted. For example, the color conversion processperformed on the first type of object image may set the target color toa color that is easy to observe as compared with the color conversionprocess performed on the second type of object image. Alternatively, thehighlight process may not be performed on the second type of objectimage.

Information about a blood vessel in a deep area is important fordiagnosing an esophageal varix. Therefore, the second type of objectimage may be preferentially highlighted when observing the gullet.

At least one of the first image and the second image may be captured byan imaging device provided outside the image processing device. In thiscase, an organ to which the object image belongs is determined based oninformation about the imaging device.

This makes it possible to use the information about the imaging deviceas the organ determination means. The imaging device corresponds to aninsertion section (scope) of an endoscope, for example. Specificexamples of the insertion section (scope) of the endoscope include anupper gastrointestinal scope, a lower gastrointestinal scope, and thelike. Since a specific identification number is assigned to each scope,each scope can be specified by storing the identification number in amemory, for example. Since a different scope is used depending on theorgan to be observed, the organ to be observed can be specified byspecifying the scope.

A plurality of object images may be present in the second image. In thiscase, the type determination section 343 determines one type among firstto Nth (N is an integer equal to or larger than 2) types to which eachof the plurality of object images belongs. The highlight section 350determines the method of performing the highlight process correspondingto the determination result.

This makes it possible to appropriately determine the type of each of aplurality of object images detected at the same time, and determine themethod of performing the highlight process corresponding to thedetermination result.

More specifically, when two blood vessels and a mucous membrane(background) are observed (see A1 in FIG. 1), three object images arepresent in total. In this case (N=3), the type determination section 343determines one type among the first to third types to which each of thethree object images belongs. As indicated by A3 in FIG. 1, the leftblood vessel that is a blood vessel in a surface area is determined tobelong to the first type, the right blood vessel that is a blood vesselin a deep area is determined to belong to the second type, and themucous membrane is determined to belong to the third type. Although anexample in which one object image corresponds to one type has beendescribed above, a person having ordinary skill in the art would readilyappreciate that a plurality of object images may correspond to one type,or no object image may correspond to a certain type.

The highlight section 350 determines the method of performing thehighlight process corresponding to the type to which each object imagebelongs. For example, a color conversion process using a differentparameter may be performed on the object image that belongs to the firsttype and the object image that belongs to the second type, and thehighlight process may not be performed on the object image that belongsto the third type (see A6 in FIG. 1).

A priority corresponding to each type may be set to each of the first toNth types. The method of performing the highlight process may bedetermined based on the priority. The highlight section 350 may set thepriority corresponding to an organ to which each object image belongs.

This makes it possible to implement the highlight process based on thepriority. The priority may be determined based on an organ to which eachobject image belongs. Specifically, information about a blood vessel ina surface area is important when observing the large intestine, andinformation about a blood vessel in a deep area is important whenobserving the gullet. Therefore, a high priority may be set to the typeto which a blood vessel in a surface area belongs (i.e., the first typein the example illustrated in FIG. 1) when observing the largeintestine, and a high priority may be set to the type to which a bloodvessel in a deep area belongs (i.e., the second type in the exampleillustrated in FIG. 1) when observing the gullet.

The highlight section 350 may perform the highlight process on an objectimage that belongs to a type with the highest priority using a highlightmethod corresponding to the type with the highest priority.

This makes it possible to cause an object image that belongs to the typewith the highest priority to relatively stand out by performing thehighlight process on only the object image that belongs to the type withthe highest priority, and not performing the highlight process on theobject image that belongs to another type.

The highlight section 350 may perform the highlight process on an objectimage that belongs to an ith type among the first to Nth types using anith highlight method, and may perform the highlight process on an objectimage that belongs to a jth type among the first to Nth types using ajth highlight method.

This makes it possible to perform the highlight process on object imagesthat respectively belong to two or more types among the first to Nthtypes using the corresponding highlight method. This is effective wheninformation about a blood vessel in a surface area and information abouta blood vessel in a deep area are important for diagnosis, for example.

The second image may be classified into a plurality of types. In thiscase, the method of performing the highlight process is determinedcorresponding to the type of the object image and the type of the secondimage. More specifically, when the second image is an NBI image, and theobject image is a blood vessel, the highlight process is performed on acorresponding attention area that is an area corresponding to a bloodvessel (see FIG. 1).

The term “corresponding attention area” used herein refers to an areathat is selected by the area selection section 344 within the firstimage, and corresponds to the attention area (i.e., an area includingthe object image) within the second image.

This makes it possible to determine the method of performing thehighlight process taking account of the type of the object image and thetype of the second image. For example, a blood vessel may be highlightedwhen the object image is a blood vessel and the second image is an NBIimage (see FIG. 1).

The feature quantity may be an edge quantity, hue, intensity, or thelike.

This makes it possible to determine the type of the object image usingan arbitrary feature quantity corresponding to the object and the like.

The type of the object image may be at least one type among the type oflesion and the type of blood vessel.

This makes it possible to determine whether or not the object image is ablood vessel, whether a blood vessel is a blood vessel in a surface areaor a blood vessel in a deep area, or whether or not the object image isa lesion.

The specific wavelength band may be narrower than the wavelength band ofthe white light. Specifically, the first image and the second image maybe an in vivo image, and the specific wavelength band may be thewavelength band of light absorbed by hemoglobin in blood. Morespecifically, the specific wavelength band may be 390 to 445 nm or 530to 550 nm.

This makes it possible to observe the structure of a blood vessel in asurface area and a deep area of tissue. A lesion area (e.g., epidermoidcancer) that cannot be easily observed using normal light can bedisplayed as a brown area or the like by inputting the resulting signalto a given channel (R, G, or B), so that the lesion area can be reliablydetected (i.e., a situation in which the lesion area is missed can beprevented). A wavelength band of 390 to 445 nm or 530 to 550 nm isselected from the viewpoint of absorption by hemoglobin and the abilityto reach a surface area or a deep area of tissue. Note that thewavelength band is not limited thereto. For example, the lower limit ofthe wavelength band may decrease by about 0 to 10%, and the upper limitof the wavelength band may increase by about 0 to 10%, depending on avariation factor (e.g., experimental results for absorption byhemoglobin and the ability to reach a surface area or a deep area oftissue).

The first image and the second image may be an in vivo image. Thespecific wavelength band included in the in vivo image may be thewavelength band of fluorescence emitted from a fluorescent substance.Specifically, the specific wavelength band may be 490 to 625 nm.

This makes it possible to implement autofluorescence imaging (AFI).Intrinsic fluorescence produced by a fluorescent substance (e.g.,collagen) can be observed by applying excitation light (390 to 470 nm).In this case, a lesion area can be highlighted in a color differing fromthat of a normal mucous membrane, so that the lesion area can bereliably detected, for example. A wavelength band of 490 to 625 nm isthe wavelength band of fluorescence produced by a fluorescent substance(e.g., collagen) when excitation light is applied. Note that thewavelength band is not limited thereto. For example, the lower limit ofthe wavelength band may decrease by about 0 to 10%, and the upper limitof the wavelength band may increase by about 0 to 10% depending on avariation factor (e.g., experimental results for the wavelength band offluorescence emitted from a fluorescent substance). A pseudo-color imagemay be generated by simultaneously applying light within a wavelengthband (540 to 560 nm) that is absorbed by hemoglobin.

The first image and the second image may be an in vivo image. Thespecific wavelength band included in the in vivo image may be thewavelength band of infrared light. Specifically, the specific wavelengthband may be 790 to 820 nm or 905 to 970 nm.

This makes it possible to implement infrared imaging (IRI). Informationabout a blood vessel or a blood flow in a deep area of a mucous membranethat is difficult to observe visually, can be highlighted byintravenously injecting indocyanine green (ICG) (infrared marker) thateasily absorbs infrared light, and applying infrared light within theabove wavelength band, so that the depth of gastric cancer invasion orthe therapeutic strategy can be determined, for example. An infraredmarker exhibits maximum absorption in a wavelength band of 790 to 820nm, and exhibits minimum absorption in a wavelength band of 905 to 970nm. Note that the wavelength band is not limited thereto. For example,the lower limit of the wavelength band may decrease by about 0 to 10%,and the upper limit of the wavelength band may increase by about 0 to10% depending on a variation factor (e.g., experimental results forabsorption by the infrared marker).

The first embodiment may also be applied to an electronic apparatus thatincludes an image processing device (image processing section).

For example, the image processing device according to the firstembodiment may be provided in various types of electronic apparatus(i.e., an apparatus that operates using a power source (e.g., voltage orcurrent)), such as an endoscope, a digital camera, a digital videocamera, and a personal computer.

The first embodiment may also be applied to a program that causes acomputer to function as a first image acquisition section, a secondimage acquisition section, the type determination section 343, and thehighlight section 350. The first image acquisition section acquires afirst image that is an image that includes an object image includinginformation within the wavelength band of white light, and the secondimage acquisition section acquires a second image that is an image thatincludes an object image including information within the specificwavelength band. The type determination section 343 determines the typeof an object image based on the feature quantity of each pixel includedin the second image. The highlight section 350 performs the highlightprocess on the first image based on the type of the object imagedetermined by the area type determination section 340.

This makes it possible to store image data in advance (e.g., capsuleendoscope), and process the stored image data by software using acomputer system (e.g., PC).

The first embodiment may also be applied to a program that causes acomputer to function as a first image acquisition section, a secondimage acquisition section, and the highlight section 350. The firstimage acquisition section acquires a first image that is an image thatincludes an object image including information within the wavelengthband of the white light, and the second image acquisition sectionacquires a second image that is an image that includes an object imageincluding information within the specific wavelength band. The highlightsection 350 performs the highlight process on the first image based onthe type of the second image.

This makes it possible to store image data in advance (e.g., capsuleendoscope), and process the stored image data by software using acomputer system (e.g., PC). This makes it possible to perform thehighlight process based on the type of the second image instead of thetype of the object image.

The first embodiment may also be applied to an image processing methodincluding acquiring a first image that is an image that includes anobject image including information within the wavelength band of thewhite light, acquiring a second image that is an image that includes anobject image including information within the specific wavelength band,determining the type of an object image based on the feature quantity ofeach pixel included in the second image, and performing the highlightprocess on the first image based on the type of the object image.

This makes it possible to implement an image processing method that canimplement the process according to the first embodiment.

The first embodiment may also be applied to an image processing methodincluding acquiring a first image that is an image that includes anobject image including information within the wavelength band of thewhite light, acquiring a second image that is an image that includes anobject image including information within the specific wavelength band,and performing the highlight process on the first image based on thetype of the second image.

This makes it possible to implement an image processing method that canimplement the process according to the first embodiment. In particular,it is possible to perform the highlight process based on the type of thesecond image instead of the type of the object image.

The first embodiment may also be also be applied to a computer programproduct that stores a program code that implements each section (e.g.,first image acquisition section, second image acquisition section, areatype determination section, and highlight section) according to thefirst embodiment.

The program code implements a first image acquisition section thatacquires a first image that is an image that includes an object imageincluding information within the wavelength band of the white light, asecond image acquisition section that acquires a second image that is animage that includes an object image including information within thespecific wavelength band, an area type determination section thatdetermines the type of an object image in the second image based on thefeature quantity of each pixel included in the second image, and ahighlight section that performs the highlight process on a signalincluded in the first image based on the type of the object imagedetermined by the area type determination section.

The term “computer program product” refers to an information storagemedium, a device, an instrument, a system, or the like that stores aprogram code, such as an information storage medium (e.g., optical diskmedium (e.g., DVD), hard disk medium, and memory medium) that stores aprogram code, a computer that stores a program code, or an Internetsystem (e.g., a system including a server and a client terminal), forexample. In this case, each element and each process according to thefirst embodiment are implemented by corresponding modules, and a programcode that includes these modules is recorded in the computer programproduct.

2. Second Embodiment

An outline of a second embodiment is described below with reference toFIG. 15. An object of the second embodiment is to determine the type ofan object image in the second image, and perform the highlight processon the first image corresponding to the type of object image in the samemanner as in the first embodiment.

In the second embodiment, the second image is an NBI image (see B1). Thetype of an object image in the second image is determined (B2). In theexample illustrated in FIG. 15, the hue H is used as the featurequantity for determining the type of the object image. An object imagehaving a hue H of 5 to 35 and having a size equal to or larger than agiven threshold value is determined to be a first type of object image,and an object image other than the first type of object image isdetermined to be a second type of object image. The first type of objectimage corresponds to a lesion area, and the second type of object imagecorresponds to an area (normal area) other than a lesion area.

A highlight range is determined after determining the type of the objectimage. In the second embodiment, the highlight process is performed onthe entire first image on condition that the first type of object image(lesion area) has been detected (B4). Therefore, a range indicated by B5is highlighted in the first image. The highlight process is notperformed when the first type of object image has not been detected.

The highlight process is performed on the highlight range using a givenmethod. In the second embodiment, the highlight process is implementedby adding a high-frequency component. More specifically, the highlightprocess is implemented using a wavelet transform or the like (B6). Thehighlight process corresponding to the type of object image can beperformed by changing the highlight process depending on whether or notthe first type of object image has been detected. A normal light imagein which the blood vessel and the lesion area are highlighted (see B7)can thus be acquired. Note that the addition of a high-frequencycomponent corresponds to highlighting the blood vessel. Since bloodvessels are densely present in a lesion area, the lesion area is alsohighlighted.

An endoscope system according to the second embodiment is describedbelow. FIG. 2 illustrates the configuration of the endoscope systemaccording to the second embodiment. The process other than the processof the area type determination section 340 and the process of thehighlight section 350 is performed in the same manner as in the firstembodiment.

A specific configuration of the area type determination section 340according to the second embodiment is described below. The area typedetermination section 340 according to the second embodiment isbasically configured in the same manner as the area type determinationsection 340 illustrated in FIG. 9. An element identical with that of thearea type determination section 340 illustrated in FIG. 9 is indicatedby an identical name and an identical reference number. The followingdescription focuses on the differences from the area type determinationsection 340 illustrated in FIG. 9.

FIG. 16 is a block diagram illustrating an example of the configurationof the area type determination section 340 according to the secondembodiment. The area type determination section 340 includes a localarea setting section 341, a feature quantity calculation section 342,and a type determination section 343. The narrow-band light image outputfrom the special light image acquisition section 330 is input to thelocal area setting section 341. The control section 360 is connected tothe local area setting section 341, the feature quantity calculationsection 342, and the type determination section 343, and controls thelocal area setting section 341, the feature quantity calculation section342, and the type determination section 343. The type determinationsection 343 is connected to the highlight section 350. The process ofthe local area setting section 341 and the process of the featurequantity calculation section 342 are the same as those described inconnection with the first embodiment.

The type determination section 343 determines whether or not each localarea is likely to be a lesion using the hue H of each local areacalculated by the feature quantity calculation section 342, and outputsthe determination result to the highlight section 350. In thenarrow-band light image used in the second embodiment, a lesion area(e.g., epidermoid cancer) is visualized (drawn) as a brown area.Therefore, whether or not each local area is likely to be a lesion canbe determined using the hue H as the feature quantity. Morespecifically, an area having a hue H of 5 to 35 may be determined to bean area that is likely to be a lesion.

The type determination section 343 outputs flag information to thehighlight section 350, the flag information indicating whether or not anarea that is likely to be a lesion is present in the narrow-band lightimage. For example, the flag information may be set to “1” when thenumber of local areas that are likely to be a lesion is equal to orlarger than a given number, and may be set to “0” when an area that islikely to be a lesion is not present.

Note that the local area set by the local area setting section 341 isnot limited to a 5×5 pixel rectangular area (refer to the firstembodiment). In the second embodiment, a relatively large area (e.g.,16×16 pixel area) may be set as the local area. This is because bloodvessels are densely present in a lesion area as compared with a normalarea. Specifically, a small area (e.g., 5×5 pixel area) may have a hue Hof 5 to 35 even if blood vessels are not densely present (e.g., even ifone blood vessel is present). Therefore, it is necessary to determinewhether or not an area in the image is a lesion based on whether or notareas that are likely to be a lesion are densely present. Specifically,it is necessary to set an appropriate threshold value, and determinewhether or not an area that is likely to be a lesion has a size equal toor larger than the threshold value.

On the other hand, when setting a relatively large area (e.g., 16×16pixel area) as the local area, the local area can include a number ofblood vessels. Therefore, only a local area in which blood vessels aredensely present has a hue H of 5 to 35. This makes it possible todetermine that an area in the image is a lesion when one area in theimage is likely to be a lesion.

The configuration of the highlight section 350 is described below. FIG.17 is a block diagram illustrating an example of the configuration ofthe highlight section 350 according to the second embodiment. Thehighlight section 350 includes a YC separation section 351, ahigh-frequency component calculation section 352, a high-frequencycomponent addition section 353, and a YC synthesis section 354. Thenormal light image output from the normal light image acquisitionsection 320 is input to the YC separation section 351. The flaginformation output from the area type determination section 340 is inputto the high-frequency component addition section 353. The controlsection 360 is connected to the YC separation section 351, thehigh-frequency component calculation section 352, the high-frequencycomponent addition section 353, and the YC synthesis section 354, andcontrols the YC separation section 351, the high-frequency componentcalculation section 352, the high-frequency component addition section353, and the YC synthesis section 354. The YC synthesis section 354 isconnected to the display section 400.

The YC separation section 351 converts the normal light image input fromthe normal light image acquisition section 320 into luminance/colordifference signals using a known YC separation process. The normal lightimage may be converted into the luminance/color difference signals usingthe following expression (11), for example. Note that Y in theexpression (11) is the luminance signal of the normal light image, andCb and Cr in the expression (11) are the color difference signals of thenormal light image.

The YC separation section 351 outputs the normal light image convertedinto the luminance/color difference signals to the high-frequencycomponent calculation section 352 and the high-frequency componentaddition section 353. Note that only the luminance signals are output tothe high-frequency component calculation section 352, and the luminancesignals and the color difference signals are output to thehigh-frequency component addition section 353.

$\begin{matrix}{\begin{bmatrix}Y \\{Cb} \\{Cr}\end{bmatrix} = {\begin{bmatrix}0.2126 & 0.7152 & 0.0722 \\{- 0.1146} & 0.3854 & 0.5000 \\0.5000 & {- 0.4542} & {- 0.0458}\end{bmatrix}\begin{bmatrix}R \\G \\B\end{bmatrix}}} & (11)\end{matrix}$

The high-frequency component calculation section 352 subjects theluminance signals of the normal light image to multiple resolutiontransformation using a known wavelet transform, extracts ahigh-frequency component from the luminance signals subjected tomultiple resolution transformation, and outputs the high-frequencycomponent to the high-frequency component addition section 353. Theprocess of the high-frequency component calculation section 352 isdescribed later.

The high-frequency component addition section 353 adds thehigh-frequency component output from the high-frequency componentcalculation section 352 to the luminance signals of the normal lightimage output from the YC separation section 351 corresponding to theflag information output from the area type determination section 340,and outputs the resulting normal light image to the YC synthesis section354.

The YC synthesis section 354 performs a known YC synthesis process onthe normal light image output from the high-frequency component additionsection 353 to obtain an RGB image, and outputs the RGB image to thedisplay section 400. The YC synthesis process may be implemented usingthe following expression (12), for example.

$\begin{matrix}{\begin{bmatrix}R \\G \\B\end{bmatrix} = {\begin{bmatrix}1.0000 & 0.0000 & 1.5748 \\1.0000 & {- 0.1873} & {- 0.4681} \\1.0000 & 1.8556 & {- 0.0458}\end{bmatrix}\begin{bmatrix}Y \\{Cb} \\{Cr}\end{bmatrix}}} & (12)\end{matrix}$

The details of the process of the high-frequency component calculationsection 352 are described below. FIG. 18 is a view illustrating anexample of the configuration of the high-frequency component calculationsection 352 according to the second embodiment. The high-frequencycomponent calculation section 352 includes a multiple resolutiontransformation section 3521 and a high-frequency component extractionsection 3522. The control section 360 is connected to the multipleresolution transformation section 3521 and the high-frequency componentextraction section 3522, and controls the multiple resolutiontransformation section 3521 and the high-frequency component extractionsection 3522. The high-frequency component extraction section 3522 isconnected to the high-frequency component addition section 353. Thenormal light image output from the YC separation section 351 is input tothe multiple resolution transformation section 3521.

The multiple resolution transformation section 3521 subjects theluminance signals of the normal light image output from the YCseparation section 351 to multiple resolution transformation using aknown wavelet transform. The image obtained by multiple resolutiontransformation is referred to as a multiple resolution-transformedimage. The wavelet transform is shown by the following expression (13)using the luminance signal Y(x, y) of the normal light image at thecoordinates (x, y) and the signal value S(i, j) of the multipleresolution-transformed image at the coordinates (i, j). Note that ImW isthe width of the normal light image, and ImH is the height of the normallight image.

$\begin{matrix}\begin{matrix}{{S\left( {i,j} \right)} = {\begin{pmatrix}{{Y\left( {{2i},{2j}} \right)} + {Y\left( {{{2i} + 1},{2j}} \right)} +} \\{{Y\left( {{2i},{{2j} + 1}} \right)} + {Y\left( {{{2i} + 1},{{2j} + 1}} \right)}}\end{pmatrix}/4}} & {\left( {{0 \leq i < {{Im}_{w}/2}},{0 \leq j < {{Im}_{H}/2}}} \right)} \\{\begin{pmatrix}{{Y\left( {{{2i} - {Im}_{w}},{2j}} \right)} - {Y\left( {{{2i} + 1 - {Im}_{w}},{2j}} \right)} +} \\{{Y\left( {{{2i} - {Im}_{w}},{{2j} + 1}} \right)} - {Y\left( {{{2i} + 1 - {Im}_{w}},{{2j} + 1}} \right)}}\end{pmatrix}/4} & {\left( {{{{Im}_{w}/2} \leq i < {Im}_{w}},{0 \leq j < {{Im}_{H}/2}}} \right)} \\{\begin{pmatrix}{{Y\left( {{2i},{{2j} - {Im}_{H}}} \right)} + {Y\left( {{{2i} + 1},{{2j} - {Im}_{H}}} \right)} -} \\{{Y\left( {{2i},{{2j} + 1 - {Im}_{H}}} \right)} - {Y\left( {{{2i} + 1},{{2j} + 1 - {Im}_{H}}} \right)}}\end{pmatrix}/4} & {\left( {{0 \leq i < {{Im}_{w}/2}},{{{Im}_{H}/2} \leq j < {Im}_{H}}} \right)} \\{\begin{pmatrix}{{Y\left( {{{2i} - {Im}_{w}},{{2j} - {Im}_{H}}} \right)} - {Y\left( {{{2i} + 1 - {Im}_{w}},{{2j} - {Im}_{H}}} \right)} -} \\{{Y\left( {{{2i} - {Im}_{w}},{{2j} + 1 - {Im}_{H}}} \right)} + {Y\left( {{{2i} + 1 - {Im}_{w}},{{2j} + 1 - {Im}_{H}}} \right)}}\end{pmatrix}/4} & {\left( {{{{Im}_{w}/2} \leq i < {Im}_{w}},{{{Im}_{H}/2} \leq j < {Im}_{H}}} \right)}\end{matrix} & (13)\end{matrix}$

FIG. 19 is a view illustrating the relationship between the multipleresolution-transformed image and the frequency component of theluminance signal of the normal light image. As illustrated in FIG. 19,the upper left area of the multiple resolution-transformed imagecorresponds to a low-frequency component of the luminance signal of thenormal light image, the upper right area of the multipleresolution-transformed image corresponds to a horizontal high-frequencycomponent of the luminance signal of the normal light image, the lowerleft area of the multiple resolution-transformed image corresponds to avertical high-frequency component of the luminance signal of the normallight image, and the lower right area of the multipleresolution-transformed image corresponds to a diagonal high-frequencycomponent of the luminance signal of the normal light image.

FIG. 20 illustrates an example of the luminance signals of an endoscopicimage, and FIG. 21 is a view illustrating the results obtained whensubjecting the luminance signals of the endoscopic image illustrated inFIG. 20 to the wavelet transform. An endoscopic image is characterizedin that a blood vessel area includes a low-frequency component and ahigh-frequency component, while a mucous membrane area includes a largeamount of a low-frequency component, but includes a small amount of ahigh-frequency component. Therefore, the multiple resolution-transformedimage illustrated in FIG. 21 has a configuration in which a blood vesselarea and a mucous membrane area are distributed in the low-frequencycomponent area illustrated in FIG. 19, and a blood vessel area is mainlydistributed in the horizontal high-frequency component area, thevertical high-frequency component area, and the diagonal high-frequencycomponent area illustrated in FIG. 19.

The high-frequency component extraction section 3522 generates ahigh-frequency component image obtained by extracting only thehigh-frequency component from the multiple resolution-transformed imageoutput from the multiple resolution transformation section 3521 using amethod described below, and outputs the high-frequency component imageto the high-frequency component addition section 353. Note that thehigh-frequency component image has the same size as that of the normallight image, and is fowled by an arbitrary combination of the horizontalhigh-frequency component, the vertical high-frequency component, and thediagonal high-frequency component of the multiple resolution-transformedimage. The signal value H(x, y) of the high-frequency component image atthe coordinates (x, y) is given by the following expression (14), forexample. Note that Floor(a) in the expression (14) indicates a processthat rounds the real number a off to the closest whole number, andabs_max(a, b, c) in the expression (14) indicates a process that selectsa real number having the maximum absolute value from the real numbers a,b, and c.

$\begin{matrix}{{H\left( {x,y} \right)} = {{abs\_ max}\begin{pmatrix}{{S\left( {{{{floor}\left( {x/2} \right)} + {{Im}_{W}/2}},{{floor}\left( {y/2} \right)}} \right)},} \\{{S\left( {{{floor}\left( {x/2} \right)},{{{floor}\left( {y/2} \right)} + {{Im}_{H}/2}}} \right)},} \\{S\left( {{{{floor}\left( {x/2} \right)} + {{Im}_{W}/2}},{{{floor}\left( {y/2} \right)} + {{Im}_{H}/2}}} \right)}\end{pmatrix}}} & (14)\end{matrix}$

The high-frequency component addition section 353 performs the highlightprocess on the normal light image output from the YC separation section351 corresponding to the flag information output from the area typedetermination section 340, and outputs the resulting normal light imageto the YC synthesis section 354.

More specifically, the high-frequency component addition section 353adds the high-frequency component image output from the high-frequencycomponent calculation section 352 to the luminance signals of the normallight image when the flag information output from the area typedetermination section 340 is set to “1” (i.e., when an area that islikely to be a lesion is present), and outputs the resulting normallight image to the YC synthesis section 354. The high-frequencycomponent addition section 353 directly outputs the normal light imageoutput from the YC separation section 351 to the YC synthesis section354 when the flag information output from the area type determinationsection 340 is set to “0” (i.e., when an area that is likely to be alesion is not present).

Since a blood vessel area that is important for diagnosing a lesion ishighlighted by performing the above process when an area that is likelyto be a lesion area is displayed in the normal light image, it ispossible to prevent a situation in which a lesion area is missed whilereducing the burden on the doctor during a diagnosis that utilizes thenormal light image and the narrow-band light image.

Although an example in which each section of the image processingsection 300 is implemented by hardware has been described above, a CPUmay perform the process of each section on an image acquired in advancein the same manner as in the first embodiment. Specifically, the processof each section may be implemented by software by causing the CPU toexecute a program. Alternatively, the process of each section maypartially be implemented by software.

A process performed on the normal light image and the narrow-band lightimage acquired in advance when implementing the process of the area typedetermination section 340 and the process of the highlight section 350illustrated in FIG. 2 by software is described below. In this case, theprocess is performed in the same manner as in the first embodimentexcept for the area type determination step (Step 3) and the highlightstep (Step 4) (see FIG. 13).

A specific process of the area type determination step (Step 3) (seeFIG. 13) according to the second embodiment is described below using aflowchart illustrated in FIG. 22. In the local area setting step, aplurality of local areas are set within the narrow-band light image inthe same manner as in the first embodiment (Step 311). In the featurequantity calculation step, the feature quantity of each local area iscalculated in the same manner as in the first embodiment (Step 312). Inthe second embodiment, the hue H is used as an example of the featurequantity. In the type determination step, the type determination processis performed on each local area, and the flag information that indicateswhether or not an area that is likely to be a lesion is present isoutput (Step 313). For example, the type determination processdetermines a local area having a hue H of 5 to 35 to be an area that islikely to be a lesion. The flag information “1” is output when thenumber of local areas that are likely to be a lesion is equal to orlarger than a given threshold value, and the flag information “0” isoutput when the number of local areas that are likely to be a lesion isequal to or less than the threshold value.

A specific process of the highlight step (Step 4) is described belowusing a flowchart illustrated in FIG. 23. In the YC separation step, thenormal light image is converted into the luminance/color differencesignals using the YC separation process shown by the expression (11)(Step 411). In the multiple resolution transformation step, the multipleresolution transformation process is performed on the luminance signalssubjected to the YC separation process using the wavelet transform shownby the expression (13) (Step 412). In the high-frequency component imagegeneration step, the high-frequency component image is generated fromthe image obtained by the multiple resolution transformation using theexpression (14) (Step 413). In Step 414, a process of Step 415 isperformed when the flag information output in Step 313 is “1”, and aprocess of Step 416 is performed when the flag information output inStep 313 is “0”. In Step 415, the high-frequency component imagegenerated in Step 413 is added to the luminance signals of the normallight image. In Step 416, the normal light image is converted into RGBsignals by performing the YC synthesis process shown by the expression(12), and the RGB signals (image) are output.

Since a blood vessel area that is important for diagnosing a lesion ishighlighted by performing the above process when an area that is likelyto be a lesion area is displayed in the normal light image, it ispossible to prevent a situation in which a lesion area is missed whilereducing the burden on the doctor during a diagnosis that utilizes thenormal light image and the narrow-band light image.

According to the second embodiment, a plurality of types may be providedas the type of the second image. In this case, the method of performingthe highlight process is determined corresponding to the type of theobject image and the type of the second image. More specifically, whenthe second image is an NBI image, and the object image is a lesion, thehighlight process is performed on the entire image (see FIG. 15).

This makes it possible to determine the method of performing thehighlight process taking account of the type of the object image and thetype of the second image. For example, the entire image may behighlighted when the object image is a lesion and the second image is anNBI image (see FIG. 15).

The highlight section 350 may enhance a specific frequency component ofthe spatial frequency of the first image over the entirety of the firstimage.

This makes it possible to enhance the specific frequency component fromthe multiple resolution-transformed image obtained by a wavelettransform or the like. For example, it is possible to highlight a bloodvessel area that includes a large amount of high-frequency component byenhancing (adding) a high-frequency component.

3. Third Embodiment

An outline of a third embodiment of the invention is described belowwith reference to FIG. 24. An object of the third embodiment is todetermine the type of an object image in the second image, and performthe highlight process on the first image corresponding to the type ofobject image in the same manner as in the first embodiment.

In the third embodiment, the second image is an NBI image (see C1). Thetype of an object image in the second image is determined (C2). In theexample illustrated in FIG. 24, the edge quantity E is used as thefeature quantity for determining the type of the object image. An objectimage having an edge quantity E equal to or larger than a giventhreshold value is determined to be a first type of object image, and anobject image other than the first type of object image is determined tobe a second type of object image. The first type of object imagecorresponds to a blood vessel, and the second type of object imagecorresponds to an area (e.g., mucous membrane) other than a bloodvessel.

A highlight range is determined after determining the type of the objectimage. In the third embodiment, the highlight process is performed on anarea of the first image that corresponds to the first type of objectimage (blood vessel) (C4). Therefore, a range indicated by C5 ishighlighted in the first image.

The highlight process is performed on the highlight range using a givenmethod. In the third embodiment, the highlight process is implemented byadding the edge quantity E. More specifically, the edge quantity E isadded to the luminance component of the first image (C6). The highlightprocess corresponding to the type of the object image can be performedby performing the highlight process on the first type of object imagewithout performing the highlight process on the second type of objectimage. A normal light image in which the blood vessel is highlighted(see C7) can thus be acquired.

An endoscope system according to the third embodiment is describedbelow. FIG. 2 illustrates the configuration of the endoscope systemaccording to the third embodiment. The process other than the process ofthe area type determination section 340 and the process of the highlightsection 350 is performed in the same manner as in the first embodiment.

A specific configuration of the area type determination section 340according to the third embodiment is described below. The area typedetermination section 340 according to the third embodiment is basicallyconfigured in the same manner as the area type determination section 340illustrated in FIG. 9. An element identical with that of the area typedetermination section 340 illustrated in FIG. 9 is indicated by anidentical name and an identical reference number. The followingdescription focuses on the differences from the area type determinationsection 340 illustrated in FIG. 9.

FIG. 25 is a block diagram illustrating an example of the configurationof the area type determination section 340 according to the thirdembodiment. The area type determination section 340 includes a luminancesignal calculation section 345, a feature quantity calculation section342, and a type determination section 343. The image signal output fromthe special light image acquisition section 330 is input to theluminance signal calculation section 345. The luminance signalcalculation section 345 is connected to the feature quantity calculationsection 342, and the feature quantity calculation section 342 isconnected to the type determination section 343. The type determinationsection 343 is connected to the highlight section 350. The controlsection 360 is connected to the luminance signal calculation section345, the feature quantity calculation section 342, and the typedetermination section 343, and controls the luminance signal calculationsection 345, the feature quantity calculation section 342, and the typedetermination section 343.

The luminance signal calculation section 345 calculates the luminancesignal Y of each pixel from the narrow-band light image input from thespecial light image acquisition section 330, and outputs the luminancesignal Y to the feature quantity calculation section 342. The luminancesignal may be calculated using the expression (11), for example.

The feature quantity calculation section 342 calculates the edgequantity E of each pixel of the normal light image using the luminancesignal output from the luminance signal calculation section 345, andoutputs the edge quantity E to the type determination section 343. Theedge quantity E(x, y) of the narrow-band light image at the coordinates(x, y) may be calculated using the following expression (15), forexample.

$\begin{matrix}{{E\left( {x,y} \right)} = {{{\sum\limits_{i = {- 1}}^{1}\;{\sum\limits_{j = {- 1}}^{1}\;{C_{i,j} \times {Y\left( {{x + i},{y + j}} \right)}}}}}\begin{matrix}{C_{i,j} =} & {8\mspace{11mu}\left( {i = {j = 0}} \right)} \\\; & {{- 1}\mspace{14mu}{else}}\end{matrix}}} & (15)\end{matrix}$

The type determination section 343 determines whether or not each pixelof the narrow-band light image is a blood vessel based on the edgequantity output from the feature quantity calculation section 342. Sincean endoscopic image is characterized in that a blood vessel area has alarge edge quantity and a mucous membrane area has a small edgequantity, a blood vessel area can be determined (discriminated) usingthe edge quantity. For example, the pixel at the coordinates (x, y) maybe determined to be a blood vessel when the edge quantity E(x, y) islarger than a threshold value E_ave (see the following expression (16)).The threshold value E_ave may be the average value calculated from theedge quantity E(x, y) of each pixel of the narrow-band light image, forexample. The type determination section 343 outputs information aboutthe position and the edge quantity E(x, y) of each pixel that has beendetermined to be a blood vessel to the highlight section 350.E(x, y)>E_ave  (16)

A specific configuration of the highlight section 350 is describedbelow. The highlight section 350 is basically configured in the samemanner as the highlight section 350 illustrated in FIG. 17. An elementidentical with that of the highlight section 350 illustrated in FIG. 17is indicated by an identical name and an identical reference number. Thefollowing description focuses on the differences from the highlightsection 350 illustrated in FIG. 17.

FIG. 26 is a block diagram illustrating an example of the configurationof the highlight section 350 according to the third embodiment. Thehighlight section 350 includes a YC separation section 351, an edgeaddition section 355, and a YC synthesis section 354. The YC separationsection 351 is connected to the edge addition section 355, and the edgeaddition section 355 is connected to the YC synthesis section 354. Theedge addition section 355 is connected to the area type determinationsection 340. The edge quantity E(x, y) is output to the edge additionsection 355. The control section 360 is connected to the YC separationsection 351, the edge addition section 355, and the YC synthesis section354, and controls the YC separation section 351, the edge additionsection 355, and the YC synthesis section 354. Note that the process ofthe YC separation section 351 and the process of the YC synthesissection 354 are the same as those described above in connection with thesecond embodiment.

The edge addition section 355 performs the highlight process on thenormal light image using the position and the edge quantity E(x, y) ofeach pixel output from the area type determination section 340. Morespecifically, the edge addition section 355 adds the edge quantity E(x,y) to the luminance signal Y(x, y) of each pixel of the normal lightimage corresponding to the position output from the area typedetermination section 340. The normal light image obtained by the edgeaddition process is output to the YC synthesis section 354.

Although an example in which the luminance conversion process isperformed on the normal light image based on the edge quantity E(x, y)has been described above, the invention is not limited thereto. Forexample, a color conversion process may be performed on the normal lightimage based on the edge quantity.

Since a blood vessel area is highlighted by performing the aboveprocess, the visibility of a blood vessel is improved. This makes itpossible to prevent a situation in which a lesion area is missed whilereducing the burden on the doctor during diagnosis using the normallight image and the narrow-band light image. An example in which onlythe edge quantity is used as the feature quantity when determining ablood vessel has been described above. When the object is tissue, anirregular area (e.g., folded area) also has a large edge quantity.Therefore, the hue H (refer to the first embodiment) may be used as thefeature quantity in addition to the edge quantity when determining ablood vessel.

Although an example in which each section of the image processingsection 300 is implemented by hardware has been described above, a CPUmay perform the process of each section on an image acquired in advancein the same manner as in the first embodiment. Specifically, the processof each section may be implemented by software by causing the CPU toexecute a program. Alternatively, the process of each section maypartially be implemented by software.

A process performed on the normal light image and the narrow-band lightimage acquired in advance when implementing the process of the area typedetermination section 340 and the process of the highlight section 350illustrated in FIG. 2 by software is described below. In this case, theprocess is performed in the same manner as in the first embodimentexcept for the area type determination step (Step 3) and the highlightstep (Step 4) (see FIG. 13).

A specific process of the area type determination step (Step 3) (seeFIG. 13) according to the third embodiment is described below using aflowchart illustrated in FIG. 27. In the luminance signal calculationstep, the luminance signal of each pixel of the narrow-band light imageis calculated using the expression (11) (Step 321). In the featurequantity calculation step, the edge quantity E(x, y) is calculated usingthe expression (15) (Step 322). In the type determination step, thedetermination process shown by the expression (16) is performed on eachpixel to determine whether or not each pixel is a blood vessel. Theposition and the edge quantity E(x, y) of each pixel that has beendetermined to be a blood vessel are output (Step 323).

In the highlight step, the edge quantity E(x, y) is added to theluminance signal Y(x, y) of each pixel of the normal light imagecorresponding to the position output in the area type determination step(Step 3) (Step 4).

Since a blood vessel area is highlighted by performing the aboveprocess, the visibility of a blood vessel is improved. This makes itpossible to prevent a situation in which a lesion area is missed whilereducing the burden on the doctor during diagnosis using the normallight image and the narrow-band light image.

According to the third embodiment, the highlight section 350 enhancesthe luminance component or the color component of the correspondingattention area in the first image based on the edge quantity calculatedfrom the second image. The term “corresponding attention area” is thesame as defined above.

This makes it possible to implement the type determination process andthe highlight process using the edge quantity E as the feature quantity.More specifically, the edge quantity E may be added to the luminancecomponent (Y component of YCrCb) and the color component (R, G, or Bcomponent) in an area in which the edge quantity E is larger than theaverage value. In the third embodiment, the edge quantity E is used asthe feature quantity used for the type determination process thatdetermines whether or not each pixel is a blood vessel, and is also usedas the parameter used for the highlight process.

4. Fourth Embodiment

An endoscope system according to a fourth embodiment of the invention isdescribed below with reference to FIG. 28. Although the first to thirdembodiments have been described above taking an example in which thenormal light image and the special light image are acquired using twoimaging elements, the normal light image and the special light image maybe acquired by image processing using only a first imaging element thatincludes a Bayer color filter array, for example. The endoscope systemaccording to the fourth embodiment includes a light source section 100,an imaging section 200, an image processing section 300, a displaysection 400, and an external I/F section 500. Note that description ofthe same elements as those described in connection with the first tothird embodiments is appropriately omitted.

The light source section 100 includes a white light source 110 thatemits white light, and a condenser lens 120.

The imaging section 200 is formed to be elongated and flexible (i.e.,can be curved) so that the imaging section 200 can be inserted into abody cavity or the like. The imaging section 200 can be removed so thata different imaging section can be used depending on the organ to beobserved. The imaging section 200 is normally referred to as a scope inthe field of endoscopy. Specific examples of the scope include an uppergastrointestinal scope, a lower gastrointestinal scope, and the like.

The imaging section 200 includes a light guide fiber 210 that guideslight focused by the light source section 100, an illumination lens 220that diffuses light that has been guided by the light guide fiber 210,and illuminates an object, an objective lens 230 that focuses lightreflected by the object, and a first imaging element 250 detects thereflected light. The first imaging element 250 includes a Bayer colorfilter array that is used to capture a normal light image, for example.The color filters of the first imaging element 250 have the spectralcharacteristics illustrated in FIG. 3, for example.

The imaging section 200 includes a memory 280. The memory 280 stores anidentification number specific to each scope. The memory 280 isconnected to a control section 360. The control section 360 can specifythe scope referring to the identification number stored in the memory280. The control section 360 can also specify the organ to be observedby specifying the scope.

The image processing section 300 includes an A/D conversion section 310,a normal light image acquisition section 320, a special light imageacquisition section 330, an area type determination section 340, ahighlight section 350, and the control section 360.

The control section 360 is connected to the area type determinationsection 340 and the highlight section 350, and controls the area typedetermination section 340 and the highlight section 350.

The external I/F section 500 is an interface that allows the user toperform an input operation on the image processing device, for example.

The A/D conversion section 310 converts an analog image signal outputfrom the first imaging element into a digital image signal, and outputsthe digital image signal.

The normal light image acquisition section 320 acquires a normal lightimage from the digital image signal output from the A/D conversionsection 310. The special light image acquisition section 330 acquires aspecial light image from the digital image signal output from the A/Dconversion section 310.

The special light image acquired by the special light image acquisitionsection 330 is output to the area type determination section 340. Thenormal light image acquired by the normal light image acquisitionsection 320 is output to the highlight section 350.

The details of the normal light image acquisition section 320 aredescribed below with reference to FIG. 6. The normal light imageacquisition section 320 includes a normal light image generation section321 and a normal light image storage section 322.

The details of the special light image acquisition section 330 aredescribed below with reference to FIG. 29. The special light imageacquisition section 330 includes a special light image generationsection 331, a special light image storage section 332, a signalextraction section 333, and a matrix data setting section 334. Thespecial light image generation section 331 performs image processing onthe digital image signal input from the A/D conversion section 310 togenerate a special light image. In the fourth embodiment, the speciallight image is a narrow-band light image. The digital image signal inputto the special light image generation section 331 is the same as thedigital image signal input to the normal light image generation section321.

The narrow-band light image is generated as described below using thesignal extraction section 333, the matrix data setting section 334, andthe special light image generation section 331. Specifically, a knowninterpolation process is performed on the input digital image signal togenerate a color image that includes R, G, and B channels. The colorimage is obtained by capturing the object using the first imagingelement 250 in a state in which white light is applied to the object.The spectral reflectivity of the object at each pixel of the color imageis estimated using a known spectrum estimation technique. The details ofthe spectrum estimation technique are disclosed in paragraphs [0054] to[0065] of JP-A-2000-115553, for example. Spectral image information inwhich each pixel has the spectrum reflectivity O(λ) (λ=380 to 780) ofthe object in the range from 380 nm to 780 nm at intervals of 10 nm isthus acquired. The spectrum reflectivity at the coordinates (x, y)within the image is indicated by O(λ)x, y. The spectral emissivity ofthe white light source is indicated by E(λ), the spectral transmittanceof the optical system is indicated by L(λ), the spectral sensitivity ofthe pixel corresponding to the color filter g2 of the second imagingelement 260 (refer to the first embodiment) is indicated by g2(λ), andthe spectral sensitivity of the pixel corresponding to the color filterb2 of the second imaging element 260 is indicated by b2(λ). The signalvalues G2′(x, y) and B2′(x, y) at the coordinates (x, y) of the G2′image and the B2′ image corresponding to the G2 image and the B2 image(refer to the first embodiment) are calculated by the followingexpressions (17) and (18), respectively.G2′(x, y)=∫E(λ)·O(λ)·L(λ)·g2(λ)dλ  (17)B2′(x, y)=∫E(λ)·O(λ)·L(λ)·b2(λ)dλ  (18)

The G2′ image and the B2′ image can be acquired from the image signalobtained by the first imaging element 250 by performing the abovecalculations over the entire image.

A color image that includes R, G, and B channels is generated from theG2′ image and the B2′ image in the same manner as in the firstembodiment. For example, a color image is generated by inputting the G2image to the R channel, and inputting the B2 image to the G channel andthe B channel. The special light image generation section 331 performs awhite balance process, a grayscale transformation process, and the likeon the generated color image, and outputs the resulting color image as anarrow-band light image. The special light image storage section 332stores the special light image output from the special light imagegeneration section 331.

The process performed after the images have been acquired by the normallight image acquisition section 320 and the special light imageacquisition section 330 is the same as that described above inconnection with the first to third embodiments.

According to the fourth embodiment, the second image acquisition section(special light image acquisition section 330 in a narrow sense)generates the second image based on the first image. More specifically,the second image acquisition section includes the signal extractionsection 333 and the matrix data setting section 334. The signalextraction section 333 extracts a signal within the wavelength band ofwhite light. The matrix data setting section 334 sets matrix data forcalculating a signal within the specific wavelength band. The secondimage acquisition section calculates a signal within the specificwavelength band from the signal extracted by the signal extractionsection 333 using the matrix data to generate the second image.

Since the second image can be generated based on the first image, anendoscope system can be implemented using only one imaging element, andthe imaging section 200 can be reduced in size (see FIG. 28). Moreover,since the number of parts can be reduced, a reduction in cost can beachieved.

5. Fifth Embodiment

An outline of a fifth embodiment of the invention is described belowwith reference to FIG. 30. An object of the fifth embodiment is todetermine the type of an object image in the second image, and performthe highlight process on the first image corresponding to the type ofobject image in the same manner as in the first embodiment.

In the fifth embodiment, the second image is a fluorescent image (seeD1). The type of an object image in the second image is determined (D2).In the example illustrated in FIG. 30, a signal value (R, G, or B signalvalue) is used as the feature quantity for determining the type of theobject image. An object image having a signal value equal to or largerthan a given threshold value is determined to be a first type of objectimage, and an object image other than the first type of object image isdetermined to be a second type of object image. The first type of objectimage corresponds to a lesion area, and the second type of object imagecorresponds to an area (normal area) other than a lesion area.Specifically, since a fluorescent agent is concentrated in a specificlesion area, a lesion area has a specific color in the second image.

A highlight range is determined after determining the type of the objectimage. In the fifth embodiment, the highlight process is performed on anarea of the first image that corresponds to the first type of objectimage (lesion area) (D4). Therefore, a range indicated by D5 ishighlighted in the first image.

The highlight process is performed on the highlight range using a givenmethod. In the fifth embodiment, the highlight process is implemented byadding and subtracting the signal value. More specifically, the value“feature quantity×gain” (gain is a given parameter) is subtracted fromthe R and G components of the first image, and the value “featurequantity×gain” is added to the B component of the first image (D6). Thehighlight process corresponding to the type of the object image can beperformed by performing the highlight process on the first type ofobject image without performing the highlight process on the second typeof object image. A normal light image in which the lesion area ishighlighted (see D7) can thus be acquired.

An endoscope system according to the fifth embodiment of the inventionis described below with reference to FIG. 31. The endoscope systemaccording to the fifth embodiment includes a light source section 100,an imaging section 200, an image processing section 300, a displaysection 400, and an external I/F section 500.

The light source section 100 includes a white light source 110 thatemits white light, a condenser lens 120 that focuses light emitted fromthe light source on a light guide fiber 210, and a rotary filter 130that extracts light within a specific wavelength band from white light.

As illustrated in FIG. 32, the rotary filter 130 includes filters F1 andF2 that differ in transmittance characteristics. The filter F1 allowslight within a wavelength band of 400 to 700 nm to pass through (seeFIG. 33). Specifically, the filter F1 allows white light to passthrough. The filter F2 (comb filter) allows light within a wavelengthband of 520 to 550 nm and light within a wavelength band of 600 to 650nm to pass through (see FIG. 34). Light within a wavelength band of 520to 550 nm extracted by the filter F2 excites a fluorescent agent (e.g.,CY3) to produce fluorescence within a wavelength band of 560 to 600 nm.Light within a wavelength band of 600 to 650 nm extracted by the filterF2 excites a fluorescent agent (e.g., CY5) to produce fluorescencewithin a wavelength band of 670 to 710 nm. The fluorescent agent CY3 isspecifically accumulated in a lesion area (e.g., epidermoid cancer) thatmay generally be observed in the gullet. The fluorescent agent CY5 isspecifically accumulated in a lesion area (e.g., tumor) that maygenerally be observed in the large intestine.

The imaging section 200 is formed to be elongated and flexible (i.e.,can be curved) so that the imaging section 200 can be inserted into abody cavity or the like. The imaging section 200 can be removed so thata different imaging section can be used depending on the organ to beobserved. The imaging section 200 is normally referred to as a scope inthe field of endoscopy. Specific examples of the scope include an uppergastrointestinal scope, a lower gastrointestinal scope, and the like.

The imaging section 200 includes the light guide fiber 210 that guideslight focused by the light source section 100, an illumination lens 220that diffuses light that has been guided by the light guide fiber 210,and illuminates an object, an objective lens 230 that focuses reflectedlight from the object, a dichroic mirror 240 that divides the focusedreflected light and fluorescence into different optical paths, and afirst imaging element 250 and a second imaging element 260 that detectthe reflected light divided by the dichroic mirror 240.

The first imaging element 250 is a Bayer color imaging element havingthe R, and B spectral characteristics illustrated in FIG. 3, forexample. The second imaging element 260 has a configuration in whichcolor filters g3 and r3 are disposed in a staggered arrangement (seeFIG. 35), for example. As illustrated in FIG. 36, the color filter g3has transmittance characteristics that allow light within a wavelengthof 560 to 600 nm to pass through, and the color filter r3 hastransmittance characteristics that allow light within a wavelength of670 to 710 nm to pass through, for example. These wavelength bandsrespectively correspond to the fluorescence wavelength of thefluorescent agent CY3 and the fluorescence wavelength of the fluorescentagent CY5.

The imaging section 200 includes a memory 280. The memory 280 stores anidentification number specific to each scope. The memory 280 isconnected to a control section 360. The control section 360 can specifythe scope referring to the identification number stored in the memory280. The control section 360 can also specify the organ to be observedby specifying the scope.

The image processing section 300 includes AD conversion sections 310 and311, a normal light image acquisition section 320, a special light imageacquisition section 330, an area type determination section 340, ahighlight section 350, the control section 360, and a link section 370.The control section 360 is connected to the normal light imageacquisition section 320, the special light image acquisition section330, the area type determination section 340, the highlight section 350,and the link section 370, and controls the normal light imageacquisition section 320, the special light image acquisition section330, the area type determination section 340, the highlight section 350,and the link section 370.

The control section 360 is also connected to the rotary filter 130. Therotary filter 130 causes illumination light to be applied to the object(i.e., tissue in a body cavity) while sequentially switching the filtersF1 and F2 by rotating a motor corresponding to a signal input from thecontrol section 360. The control section 360 outputs information aboutthe filter F1 or F2 disposed in the optical path to the normal lightimage acquisition section 320, the special light image acquisitionsection 330, and the link section 370 as a trigger signal.

The external I/F section 500 is an interface that allows the user toperform an input operation on the image processing device, for example.

The AD conversion section 310 converts an analog image signal outputfrom the first imaging element 250 into a digital image signal, andoutputs the digital image signal. The AD conversion section 311 convertsan analog image signal output from the second imaging element 260 into adigital image signal, and outputs the digital image signal.

The normal light image acquisition section 320 acquires a normal lightimage from the digital image signal output from the AD conversionsection 310. The special light image acquisition section 330 acquires aspecial light image from the digital image signal output from the ADconversion section 311.

The normal light image acquired by the normal light image acquisitionsection 320 and the special light image acquired by the special lightimage acquisition section 330 are output to the link section 370. In thefifth embodiment, since the normal light image and the special lightimage are alternately acquired by the normal light image acquisitionsection 320 and the special light image acquisition section 330, thelink section 370 links the normal light image and the special lightimage. The details of the link section 370 are described later.

The normal light image linked by the link section 370 is output to thehighlight section 350. The special light image linked by the linksection 370 is output to the area type determination section 340. Thearea type determination section 340 determines the type of an object inthe special light image, and outputs the determination result to thehighlight section 350. The highlight section 350 performs a highlightprocess on the normal light image corresponding to the determinationresult output from the area type determination section 340, and outputsthe resulting normal light image to the display section 400. The detailsof the area type determination section 340 and the highlight section 350are described later.

The details of the normal light image acquisition section 320 will nowbe described with reference to FIG. 6. The normal light imageacquisition section 320 includes a normal light image generation section321 and a normal light image storage section 322. The normal light imagegeneration section 321 specifies a period in which the filter F1 ispositioned in the optical path based on the trigger signal transmittedfrom the control section 360, and performs image processing on thedigital image signal converted from the analog image signal transmittedfrom the first imaging element in a period in which the filter F1 ispositioned in the optical path to generate a normal light image. Morespecifically, the normal light image generation section 321 performs aninterpolation process, a white balance process, a color conversionprocess, a grayscale transformation process, and the like on the digitalimage signal to generate a normal light image, and outputs the normallight image. The normal light image is an RGB image. The normal lightimage storage section 322 stores the normal light image output from thenormal light image generation section 321 in a memory.

The details of the special light image acquisition section 330 will nowbe described with reference to FIG. 7. The special light imageacquisition section 330 includes a special light image generationsection 331 and a special light image storage section 332. The speciallight image generation section 331 specifies a period in which thefilter F2 is positioned in the optical path based on the trigger signaltransmitted from the control section 360, and performs image processingon the digital image signal converted from the analog image signaltransmitted from the second imaging element in a period in which thefilter F2 is positioned in the optical path to generate a special lightimage. In the fifth embodiment, the special light image is a fluorescentimage. The fluorescent image is an RGB image.

The special light image generation section 331 generates the fluorescentimage as described below. The second imaging element 260 has aconfiguration in which the color filters g3 and r3 are disposed in astaggered arrangement (see FIG. 37). Therefore, digital image signalsillustrated in FIG. 37 are input to the special light image generationsection 331. Note that G3(x, y) indicates the signal value of the g3filter, and R3(x, y) indicates the signal value of the r3 filter. x andy are image coordinate values. The special light image generationsection 331 performs an interpolation process on the image signal togenerate a G3 image in which each pixel has the signal value of thefilter g3, and an R3 image in which each pixel has the signal value ofthe filter r3. The interpolation process may be performed as shown bythe expressions (1) and (2), for example.

A color image having R, G, and B signal values is generated from the G3image and the R3 image in the same manner as the normal light image. Forexample, the color image is generated by inputting the signal valueR3(x, y) to the R signal at the coordinates (x, y), and inputting thesignal value G3(x, y) to the G and B signals at the coordinates (x, y).The special light image generation section 331 performs a white balanceprocess, a grayscale transformation process, and the like on thegenerated color image, and outputs the resulting color image as afluorescent image. The special light image storage section 332 storesthe fluorescent image output from the special light image generationsection 331 in a memory.

FIG. 38 is a view illustrating the type of filter positioned in theoptical path, and images stored in the normal light image storagesection 322 and the special light image storage section 332. Asillustrated in FIG. 38, the filter F1 is inserted into the optical pathat a timing 1. In this case, white light is emitted as the illuminationlight. The normal light image is stored in the normal light imagestorage section 322, and an image is not stored in the special lightimage storage section 332. The filter F2 is inserted into the opticalpath at a timing 2. In this case, excitation light is emitted as theillumination light. Fluorescence produced from a lesion area where thefluorescent agent is accumulated is stored in the special light imagestorage section 332 as a pseudo-color image, and an image is not storedin the normal light image storage section 322. The normal light imagestorage section 322 and the special light image storage section 332 canstore a plurality of images.

The details of the link section 370 will now be described. In the fifthembodiment, since the normal light image and the special light image arealternately acquired by the normal light image acquisition section 320and the special light image acquisition section 330, the link section370 links the normal light image and the special light image. Theprocess of the link section 344 according to the fifth embodiment isdescribed in detail below.

FIG. 39 is a view illustrating a timing at which the image stored in thenormal light image storage section 322 was acquired, and a timing atwhich the image stored in the special light image storage section 332was acquired. The link section 370 sequentially reads the normal lightimage and the fluorescent image that are linked so that the differencein image acquisition timing becomes a minimum from the normal lightimage storage section 322 and the special light image storage section332 according to a control signal input from the control section 360.Specifically, the link section 370 reads the normal light image acquiredat the timing 1 and the fluorescent image acquired at the timing 2, andthen reads the normal light image acquired at the timing 2 and thefluorescent image acquired at the timing 3. The link section 370 thusacquires the normal light image and the fluorescent image at the sameinterval as the image acquisition interval.

The details of the area type determination section 340 will now bedescribed. The area type determination section 340 is configured in thesame manner as in the first embodiment (see FIG. 9). The area typedetermination section 340 performs the process in the same manner as inthe first embodiment.

The feature quantity calculation section 342 calculates the featurequantity corresponding to the organ to be observed. The fluorescentagent CY3 used in connection with the fifth embodiment is specificallyaccumulated in a lesion area (e.g., epidermoid cancer) that maygenerally be observed in the gullet, and the fluorescent agent CY5 usedin connection with the fifth embodiment is specifically accumulated in alesion area (e.g., tumor) that may generally be observed in the largeintestine. Therefore, the G or B signal to which the fluorescence signalof the fluorescent agent CY3 is input may be used as the featurequantity when observing the gullet, and the R signal to which thefluorescence signal of the fluorescent agent CY5 is input may be used asthe feature quantity when observing the large intestine. The controlsection 360 may specify the organ to be observed referring to theidentification number specific to each scope stored in the memory 280,or the user may designate the organ to be observed via the external I/Fsection 500.

An example in which the large intestine is observed (diagnosed) isdescribed below. When observing the large intestine, the featurequantity calculation section 342 calculates the average value of the Rsignal of each pixel of each local area set by the local area settingsection 341. The feature quantity calculation section 342 outputs thecalculated average value of the R signals to the type determinationsection 343 as the feature quantity. Note that the feature quantity ofthe local area a(m, n) is indicated by f(m, n).

The type determination section 343 determines the type of object in eachlocal area based on the feature quantity f(m, n) of each local areacalculated by the feature quantity calculation section 342.

Since the fluorescent agent CY5 used in connection with the fifthembodiment is specifically accumulated in a lesion area (e.g., tumor), alocal area in which a lesion area (e.g., tumor) is present tends to havea feature quantity f(m, n) larger than that of a local area in which alesion area is not present. Therefore, a local area having a featurequantity f(m, n) larger than a threshold value F_th may be determined tobe a lesion area (e.g., tumor). The threshold value F_th may be theaverage feature quantity f(m, n) of each local area (see the followingexpression (19)), for example.

$\begin{matrix}{{F\_ th} = {\frac{1}{\left( {M + 1} \right)\left( {N + 1} \right)}{\sum\limits_{m = 0}^{M}\;{\sum\limits_{n = 0}^{N}\;{f\left( {m,n} \right)}}}}} & (19)\end{matrix}$

The type determination section 343 outputs the coordinates and thefeature quantity f(m, n) of each local area that has been determined tobe a lesion area (e.g., tumor), and tag information that indicates thedetermination result to the area selection section 344. The taginformation about an area that has been determined to be a lesion area(e.g., tumor) may be set to “1”, for example.

The area selection section 344 calculates the position of each pixel ofeach local area a(m, n) that has been determined to be a lesion area(e.g., tumor) by the type determination section 343 from the coordinatesof each local area and information about each pixel included in eachlocal area, and outputs the position of each pixel thus calculated, thetag information, and the feature quantity f(x, y) to the highlightsection 350. Note that the feature quantity f(x, y) at the position ofeach pixel may be the feature quantity f(m, n) of the local area a(m, n)to which each pixel belongs.

The details of the highlight section 350 are described below. Thehighlight section 350 performs the highlight process on each pixel ofthe normal light image that corresponds to the position of each pixeloutput from the area type determination section 340. The normal lightimage subjected to the highlight process is output to the displaysection 400.

The highlight process performed on a pixel that has been determined tobe a lesion area (e.g., tumor) (i.e., tag value=“1”) may be implementedby a color conversion process shown by the following expression (20),for example.R _(—out)(x, y)=R(x, y)−gain*f(x, y)G _(—out)(x, y)=G(x, y)−gain*f(x, y)B _(—out)(x, y)=B(x, y)+gain*f(x, y)  (20)where, R(x, y), G(x, y), and B(x, y) are the R, G, and B signal valuesof the normal light image at the coordinates (x, y) before the colorconversion process is performed, and R_out(x, y), G_out(x, y), andB_out(x, y) are the R, G, and B signal values of the normal light imageat the coordinates (x, y) after the color conversion process has beenperformed.

gain is an arbitrary coefficient in the range of 0 to 1. The coefficientgain may be set by the user via the external IN section 500, or may beset in advance corresponding to the organ to be observed. The controlsection 360 may specify the organ to be observed referring to theidentification number specific to each scope stored in the memory 280,or the user may designate the organ to be observed via the external I/Fsection 500.

Since a lesion area (e.g., tumor) is drawn as a blue area differing froma normal area by performing the above process, it is possible to preventa situation in which a lesion area is missed while reducing the burdenon the doctor during diagnosis using the normal light image and thefluorescent image.

Although an example in which the color conversion process is performedon the normal light image based on the feature quantity has beendescribed above, the invention is not limited thereto. For example, anarbitrary color conversion process or a luminance conversion process maybe performed based on the feature quantity.

Although the fifth embodiment utilizes the fluorescent agent, intrinsicfluorescence produced by collagen in tissue may be observed (e.g.,autofluorescence imaging (AFI)), for example. In this case, light withina wavelength band of 390 to 470 nm may be used as excitation light, andthe transmission characteristics of the color filter of the secondimaging element may be changed to 490 to 625 nm (i.e., the wavelengthband of intrinsic fluorescence).

Light within a wavelength band of 790 to 820 nm (infrared light) andlight within a wavelength band of 905 to 970 nm (infrared light) may beused as illumination light after intravenously injecting indocyaninegreen (ICG), and a pseudo-color image may be generated from reflectedlight images, and used as the special light image (e.g., infraredimaging (IRI)).

Note that each section of the image processing section 300 need notnecessarily be implemented by hardware. For example, a CPU may performthe process of each section on an image acquired in advance in the samemanner as in the first embodiment. Specifically, the process of eachsection may be implemented by software by causing the CPU to execute aprogram. Alternatively, the process of each section may partially beimplemented by software.

A process performed on the normal light image and the fluorescent imageacquired in advance when implementing the process of the area typedetermination section 340, the process of the highlight section 350, andthe process of the link section 370 illustrated in FIG. 31 by softwareis described below using a flowchart illustrated in FIG. 40.

In Step 11, the normal light image and the fluorescent image that havebeen alternately acquired are linked based on image acquisition timinginformation. The fluorescent image is written into the memory (Step 12),and the normal light image linked to the fluorescent image is thenwritten into the memory (Step 13).

FIG. 41 illustrates the details of the area type determination step(Step 14). In the local area setting step, a plurality of local areasare set within the fluorescent image in the same manner as in the firstembodiment (Step 141). In the feature quantity calculation step, thefeature quantity of each local area is calculated corresponding to theorgan to be observed in the same manner as described above (Step 142).In the type determination step, whether or not each local area is alesion area (e.g., tumor) is determined by the threshold process (Step143). In the area election step, the position of each pixel included ineach local area that has been determined to be a lesion area iscalculated from the coordinates of each local area information abouteach pixel included in each local area (Step 144).

In the highlight step (Step 15), the highlight process is performed onthe normal light image using the method shown by the expression (19).The process ends when all of the images have been processed. The processis repeated until all of the images have been processed (Step 16).

According to the fifth embodiment, the type determination section 343changes the feature quantity calculation process corresponding to theorgan to which the object image belongs.

This makes it possible to change the feature quantity used to determinethe type of object image corresponding to the organ to be observed.Specifically, the fluorescence signal G3 of the fluorescent agent CY3 isinput to the G and B channels, and the fluorescence signal R3 of thefluorescent agent CY5 is input to the R channel during observation usingthe fluorescent agents CY3 and CY5. The fluorescent agent CY3 tends tobe specifically accumulated in a lesion area (e.g., epidermoid cancer)that may generally be observed in the gullet, and the fluorescent agentCY5 tends to be specifically accumulated in a lesion area (e.g., tumor)that may generally be observed in the large intestine. Therefore, the Gor B signal to which the fluorescence signal G3 of the fluorescent agentCY3 is input may be used as the feature quantity when observing thegullet, and the R signal to which the fluorescence signal R3 of thefluorescent agent CY5 is input may be used as the feature quantity whenobserving the large intestine.

The highlight section 350 may perform the highlight process thatenhances the luminance component or the color component of thecorresponding attention area in the first image based on the colorfeature quantity calculated from the second image.

This makes it possible to add the color feature quantity (i.e., the G orB signal when observing the gullet, and the R signal when observing thelarge intestine) to the luminance component (Y component) or the colorcomponent (R, G, or B component) of the corresponding attention area.

6. Sixth Embodiment

An outline of a sixth embodiment of the invention is described belowwith reference to FIG. 42. An object of the sixth embodiment is todetermine the type of an object image in the second image, and performthe highlight process on the first image corresponding to the type ofobject image in the same manner as in the first embodiment. Since anendoscope system according to the sixth embodiment is configured so thata plurality of second images can be acquired, it is necessary to takeaccount of the type of object image and the type of second image.

The type of second image is determined (E1). The second image may be afluorescent image (E2) or an NBI image (E3). The subsequent processdiffers depending on whether the second image is a fluorescent image oran NBI image. This makes it possible to perform the highlight process onthe first image corresponding to the type of the second image.

When the second image is a fluorescent image, the subsequent process isperformed in the same manner as illustrated in FIG. 30 (fifthembodiment). When the second image is an NBI image, the subsequentprocess is performed in the same manner as illustrated in FIG. 24 (thirdembodiment).

The endoscope system according to the sixth embodiment is now describedwith reference to FIG. 43. The endoscope system according to the sixthembodiment includes a light source section 100, an imaging section 200,an image processing section 300, a display section 400, and an externalI/F section 500.

The light source section 100 includes a white light source 110 thatemits white light, a condenser lens 120 that focuses light emitted fromthe light source on a light guide fiber 210, and a first rotary filter140 and a second rotary filter 150 that extract light within a specificwavelength band from white light.

Note that the first rotary filter 140 and the second rotary filter 150are controlled exclusively. Specifically, when the first rotary filter140 is inserted into the optical path of light emitted from the whitelight source 110, the second rotary filter 150 is not inserted into theoptical path of light emitted from the white light source 110. When thesecond rotary filter 150 is inserted into the optical path of lightemitted from the white light source 110, the first rotary filter 140 isnot inserted into the optical path of light emitted from the white lightsource 110. The user may designate the rotary filter via the external PFsection 500, and a control section 360 may exclusively control the firstrotary filter 140 and the second rotary filter 150 according toinstructions from the user, for example.

As illustrated in FIG. 44, the first rotary filter 140 includes filtersF1 and F3 that differ in transmittance characteristics. The filter F1allows light within a wavelength band of 400 to 700 nm to pass through(see FIG. 33). The filter F3 allows light within a wavelength band of600 to 650 nm to pass through (see FIG. 46). Light within a wavelengthband of 600 to 650 nm extracted by the filter F3 excites a fluorescentagent (e.g., CY5) to produce fluorescence within a wavelength band of670 to 710 nm. The fluorescent agent CY5 is specifically accumulated ina lesion area (e.g., tumor).

As illustrated in FIG. 45, the second rotary filter 150 includes filtersF1 and F4 that differ in transmittance characteristics. The filter F4(comb filter) allows light within a wavelength band of 390 to 445 nm andlight within a wavelength band of 530 to 550 nm to pass through (seeFIG. 47). The characteristics of the filter F1 included in the secondrotary filter 150 are identical with the characteristics of the filterF1 included in the first rotary filter 140.

The imaging section 200 is formed to be elongated and flexible (i.e.,can be curved) so that the imaging section 200 can be inserted into abody cavity or the like. The imaging section 200 includes the lightguide fiber 210 that guides light focused by the light source section100, an illumination lens 220 that diffuses light that has been guidedby the light guide fiber 210, and illuminates an object, an objectivelens 230 that focuses reflected light from the object, a dichroic mirror240 that divides the focused reflected light into different opticalpaths, and a first imaging element 250 and a second imaging element 260that detect the reflected light divided by the dichroic mirror 240.

The first imaging element 250 is a Bayer color imaging element havingthe R, and B spectral characteristics illustrated in FIG. 3, forexample. The second imaging element 260 has a configuration in whichcolor filters g4 and b4 are disposed in a staggered arrangement (seeFIG. 48), for example. As illustrated in FIG. 49, the color filter g4(comb filter) allows light within a wavelength band of 530 to 550 nm andlight within a wavelength band of 670 to 710 nm to pass through, forexample. As illustrated in FIG. 50, the color filter b4 (comb filter)allows light within a wavelength band of 390 to 445 nm and light withina wavelength band of 670 to 710 nm to pass through, for example. Thewavelength band of 670 to 710 nm common to the color filters g4 and b4corresponds to the fluorescence wavelength of the fluorescent agent CY5.

The image processing section 300 includes AD conversion sections 310 and311, a normal light image acquisition section 320, a special light imageacquisition section 330, a highlight section 350, the control section360, and a link section 370. The control section 360 is connected to thenormal light image acquisition section 320, the special light imageacquisition section 330, the highlight section 350, and the link section370, and controls the normal light image acquisition section 320, thespecial light image acquisition section 330, the highlight section 350,and the link section 370.

The control section 360 is also connected to the first rotary filter 140and the second rotary filter 150. The first rotary filter 140 and thesecond rotary filter 150 cause illumination light to be applied to theobject (i.e., tissue in a body cavity) while sequentially switching thefilters F1 and F3 or the filters F1 and F4 by rotating a motor accordingto a signal output from the control section 360. The control section 360outputs information about the rotary filter inserted into the opticalpath and information about the filter F1, F3, or F4 disposed in theoptical path to the normal light image acquisition section 320, thespecial light image acquisition section 330, the highlight section 350,and the link section 370. The information output from the controlsection 360 is a flag signal and a trigger signal. The flag signalindicates whether the first rotary filter 140 or the second rotaryfilter 150 is inserted into the optical path. The trigger signalindicates the filter F1, F3, or F4 that is disposed in the optical path.

The external I/F section 500 is an interface that allows the user toperform an input operation on the image processing device, for example.

The AD conversion section 310 converts an analog image signal outputfrom the first imaging element 250 into a digital image signal, andoutputs the digital image signal. The AD conversion section 311 convertsan analog image signal output from the second imaging element 260 into adigital image signal, and outputs the digital image signal.

The normal light image acquisition section 320 acquires a normal lightimage from the digital image signal output from the AD conversionsection 310. The special light image acquisition section 330 acquires aspecial light image from the digital image signal output from the ADconversion section 311.

The normal light image acquired by the normal light image acquisitionsection 320 and the special light image acquired by the special lightimage acquisition section 330 are output to the link section 370. In thesixth embodiment, since the normal light image and the special lightimage are alternately acquired by the normal light image acquisitionsection 320 and the special light image acquisition section 330, thelink section 370 links the normal light image and the special lightimage. The process of the link section 370 is the same as that describedabove in connection with the fifth embodiment.

The normal light image and the special light image linked by the linksection 370 are output to the highlight section 350. The highlightsection 350 performs the highlight process on the normal light imagecorresponding to the type of the special light image, and outputs theresulting normal light image to the display section 400. The details ofthe highlight section 350 are described later.

The details of the normal light image acquisition section 320 aredescribed below with reference to FIG. 6. The normal light imageacquisition section 320 includes a normal light image generation section321 and a normal light image storage section 322. The normal light imagegeneration section 321 specifies a period in which the filter F1 ispositioned in the optical path based on the trigger signal transmittedfrom the control section 360, and performs image processing on thedigital image signal converted from the analog image signal transmittedfrom the first imaging element in a period in which the filter F1 ispositioned in the optical path to generate a normal light image. Morespecifically, the normal light image generation section 321 performs aninterpolation process, a white balance process, a color conversionprocess, a grayscale transformation process, and the like on the digitalimage signal to generate a normal light image, and outputs the normallight image. The normal light image is an RGB image. The normal lightimage storage section 322 stores the normal light image output from thenormal light image generation section 321 in a memory. The normal lightimage storage section 322 can store a plurality of images.

The details of the special light image acquisition section 330 aredescribed below with reference to FIG. 7. The special light imageacquisition section 330 includes a special light image generationsection 331 and a special light image storage section 332. The speciallight image generation section 331 specifies whether the first rotaryfilter 140 or the second rotary filter 150 is inserted into the opticalpath based on the flag signal transmitted from the control section 360,and generates the special light image corresponding to the determinationresult as to whether the first rotary filter 140 or the second rotaryfilter 150 is inserted into the optical path. The special light imagegeneration method differs between the case where the first rotary filter140 is inserted into the optical path and the case where the secondrotary filter 150 is inserted into the optical path as described below.

When the first rotary filter 140 is inserted into the optical path, thespecial light image acquisition section 330 specifies a period in whichthe filter F3 is positioned in the optical path based on the triggersignal, and performs image processing on the digital image signal outputfrom the AD conversion section 311 in a period in which the filter F3 ispositioned in the optical path to generate a special light image. Thefilter F3 allows light within a wavelength band that excites thefluorescent agent CY5 to produce fluorescence to pass through. Thefluorescence wavelength band of the fluorescent agent CY5 is 670 to 710nm The color filters g4 and b4 used for the second imaging element 260allow light within the above fluorescence wavelength band to passthrough. Therefore, a fluorescence signal of the fluorescent agent CY5is output from the second imaging element 260. Accordingly, the speciallight image acquired when the first rotary filter 140 is inserted intothe optical path is a fluorescent image. The special light imagegeneration section 331 performs a grayscale transformation process onthe digital image signal output from the AD conversion section 311 togenerate a fluorescent image, and outputs the generated fluorescentimage to the special light image storage section 332. The fluorescentimage is a monochromatic image. The special light image storage section332 stores the fluorescent image output from the special light imagegeneration section 331 in a memory.

When the second rotary filter 150 is inserted into the optical path, thespecial light image acquisition section 330 specifies a period in whichthe filter F4 is positioned in the optical path based on the triggersignal, and performs image processing on the digital image signal outputfrom the AD conversion section 311 in a period in which the filter F4 ispositioned in the optical path to generate a special light image. Thefilter F4 allows narrow-band light within a wavelength band of 390 to445 nm and narrow-band light within a wavelength band of 530 to 550 nmto pass through. The second imaging element 260 has a configuration inwhich the color filters g4 and b4 are disposed in a staggeredarrangement (see FIG. 37), as described above. The color filter g4allows light within a wavelength band of 530 to 550 nm to pass through,and the color filter b4 allows light within a wavelength band of 390 to445 nm to pass through. Therefore, the digital image signal input to thespecial light image generation section 331 when the second rotary filter150 is inserted into the optical path is the same as the digital imagesignal input to the special light image generation section 331 in thefirst embodiment. Therefore, the special light image generation section331 generates a narrow-band light image in the same manner as in thefirst embodiment, and outputs the generated narrow-band light image tothe special light image storage section 332. The narrow-band light imageis an RGB image. The special light image storage section 332 stores thenarrow-band light output from the special light image generation section331 in a memory. The special light image storage section 332 can store aplurality of images.

The details of the highlight section 350 are described below. Asillustrated in FIG. 51, the highlight section 350 includes a featurequantity calculation section 356 and a feature quantity addition section357. The special light image that has been linked to the normal lightimage by the link section 370 is output to the feature quantitycalculation section 356. The normal light image that has been linked tothe special light image by the link section 370 is output to the featurequantity addition section 357. The control section 360 is connected tothe feature quantity calculation section 356 and the feature quantityaddition section 357, and controls the feature quantity calculationsection 356 and the feature quantity addition section 357.

The feature quantity calculation section 356 calculates the featurequantity corresponding to the type of the special light image. Morespecifically, the feature quantity calculation section 356 specifieswhether the first rotary filter 140 or the second rotary filter 150included in the light source section 100 is inserted into the opticalpath based on the flag signal transmitted from the control section 360,and calculates the feature quantity corresponding to the determinationresult as to whether the first rotary filter 140 or the second rotaryfilter 150 is inserted into the optical path.

The special light image output from the link section 370 when the firstrotary filter 140 is inserted into the optical path is a monochromaticfluorescent image. In this case, the feature quantity calculationsection 356 outputs the signal value at the position of each pixel ofthe input fluorescent image (x, y) to the feature quantity additionsection 357 as the feature quantity. Note that the feature quantity atthe pixel position (x, y) is indicated by f′(x, y).

The special light image output from the link section 370 when the secondrotary filter 150 is inserted into the optical path is a narrow-bandlight image. In this case, the feature quantity calculation section 356calculates the edge quantity E(x, y) of each pixel of the special lightimage from the luminance signals of the input narrow-band light imageusing the expression (15), and outputs the calculated edge quantity E(x,y) to the feature quantity addition section 357 as the feature quantityf′(x, y).

The feature quantity addition section 357 performs the highlight processon the normal light image corresponding to the type of the special lightimage. The type of the special light image may be determined referringto the flag signal transmitted from the control section 360.

When the first rotary filter 140 is inserted into the optical path, thefeature quantity addition section 357 performs the color conversionprocess on the normal light image using the expression (20). Although anexample in which the color conversion process is performed on the normallight image based on the feature quantity calculated by the featurequantity calculation section 356 has been described above, the inventionis not limited thereto. For example, an arbitrary color conversionprocess or a luminance conversion process may be performed based on thefeature quantity.

When the second rotary filter 150 is inserted into the optical path, thefeature quantity addition section 357 adds the feature quantity f′(x, y)output from the feature quantity calculation section 356 to theluminance signal Y(x, y) of the normal light image. Although an examplein which the luminance conversion process is performed on the normallight image based on the feature quantity calculated by the featurequantity calculation section 356 has been described above, the inventionis not limited thereto. For example, an arbitrary color conversionprocess or an arbitrary luminance conversion process may be performedbased on the feature quantity.

When a fluorescent image is acquired as the special light image byperforming the above process, a lesion area (e.g., tumor) is drawn in acolor differing from that of a normal area. When a narrow-band lightimage is acquired as the special light image by performing the aboveprocess, a blood vessel area that is important for diagnosing a lesionis highlighted.

The above configuration makes it possible to perform the highlightprocess on the normal light image corresponding to the type of thespecial light image. This makes it possible to prevent a situation inwhich a lesion area is missed while reducing the burden on the doctorduring diagnosis using the normal light image and the special lightimage.

Although an example in which the type of the special light image isdetermined using the information about the rotary filter that isinserted into the optical path as the flag signal has been describedabove, the invention is not limited thereto. For example, the type ofthe special light image may be determined based on the signal outputfrom the second imaging element 260.

A method that determines the type of the special light image based onthe signal output from the second imaging element 260 is describedbelow. As illustrated in FIG. 48, the second imaging element 260 has aconfiguration in which the color filters g4 and b4 are disposed in astaggered arrangement. When the first rotary filter 140 is inserted intothe optical path, the fluorescence signal of the fluorescent agent CY5is output from the second imaging element 260. Therefore, the averagevalue G4_ave of the signal output from each pixel that corresponds tothe color filter g4 is almost equal to the average value B4_ave of thesignal output from each pixel that corresponds to the color filter b4.

When the second rotary filter 150 is inserted into the optical path, anarrow-band signal that corresponds to a wavelength band of 530 to 550nm (color filter g4) and a narrow-band signal that corresponds to awavelength band of 390 to 445 nm (color filter b4) are output from thesecond imaging element 260 (i.e., signals that differ in wavelength bandare output depending on the color filter). Therefore, the average valuesG4_ave and B4_ave differ from each other. Accordingly, the type of thespecial light image can be determined by comparing the average valuesG4_ave and B4_ave.

Note that each section of the image processing section 300 need notnecessarily be implemented by hardware. For example, a CPU may performthe process of each section on an image acquired in advance in the samemanner as in the first embodiment. Specifically, the process of eachsection may be implemented by software by causing the CPU to execute aprogram. Alternatively, the process of each section may partially beimplemented by software.

A process performed on the normal light image and the special lightimage acquired in advance when implementing the process of the highlightsection 350 and the process of the link section 370 illustrated in FIG.43 by software is described below using a flowchart illustrated in FIG.52.

In Step 21, the normal light image and the special light image that havebeen alternately acquired are linked based on the image acquisitiontiming information in the same manner as in the fifth embodiment. Thespecial light image is written into the memory (Step 22), and the normallight image linked to the special light image is then written into thememory (Step 23). The flag signal that specifies the type of the speciallight image is then written into the memory (Step 24).

FIG. 53 illustrates the details of the highlight step (Step 25). In thespecial light image determination step, the type of the special lightimage is determined referring to the flag signal (Step 251). In thefeature quantity calculation step, the feature quantity is calculatedcorresponding to the type of the special light image (Step 252). Morespecifically, when the special light image is a fluorescent image, thesignal value of the fluorescent image is used as the feature quantityf′(x, y). When the special light image is a narrow-band light image, theedge quantity is calculated from the luminance signal of the narrow-bandlight image using the expression (15), and used as the feature quantityf′(x, y). In the feature quantity addition step, the highlight processis performed on the normal light image corresponding to the type of thespecial light image (Step 253). More specifically, when the speciallight image is a fluorescent image, the color conversion process isperformed on the normal light image using the feature quantitycalculated in Step 252 and the expression (20). When the special lightimage is a narrow-band light image, the feature quantity calculated inStep 252 is added to the luminance signal of the normal light image.

The process ends when all of the images have been processed. The processis repeated until all of the images have been processed (Step 26).

According to the sixth embodiment, the first image acquisition section(normal light image acquisition section 320 in a narrow sense) acquiresthe first image (normal light image in a narrow sense) that correspondsto the wavelength band of white light, and the second image acquisitionsection (special light image acquisition section 330 in a narrow sense)acquires the second image (special light image (e.g., narrow-band imageor fluorescent image) in a narrow sense) that corresponds to thespecific wavelength band (the wavelength band of narrow-band light orfluorescence in a narrow sense). The highlight section 350 performs thehighlight process on the first image based on the type of the secondimage.

The second image may be an NBI image or an AFI image. The highlightsection 350 may change the method of performing the highlight processdepending on whether the second image is an NBI image or an AFI image.

This makes it possible to acquire the normal light image (white lightimage) and the special light image (e.g., NBI image or fluorescentimage), and change the method of performing the highlight process on thenormal light image corresponding to the type of the special light image.Therefore, it is possible to perform the process in the same manner asin the third embodiment when the special light image is an NBI image,and perform the process in the same manner as in the fifth embodimentwhen the special light image is a fluorescent image.

The first to sixth embodiments according to the invention and themodifications thereof have been described above. Note that the inventionis not limited to the first to sixth embodiments and the modificationsthereof. Various modifications and variations may be made withoutdeparting from the scope of the invention. A plurality of elementsdescribed in connection with the first to sixth embodiments and themodifications thereof may be appropriately combined to achieve variousconfigurations. For example, some elements may be omitted from theelements described in connection with the first to sixth embodiments andthe modifications thereof. Some of the elements described in connectionwith different embodiments or modifications thereof may be appropriatelycombined. Specifically, various modifications and applications arepossible without materially departing from the novel teachings andadvantages of the invention.

Any term (e.g., normal light image or special light image) cited with adifferent term (e.g., first image or second image) having a broadermeaning or the same meaning at least once in the specification and thedrawings may be replaced by the different term in any place in thespecification and the drawings.

What is claimed is:
 1. An image processing device comprising: a firstimage acquisition section that acquires a first image that includes anobject image including information within a wavelength band of whitelight; a second image acquisition section that acquires a second imagethat includes an object image including information within a specificwavelength band; a type determination section that determines a type ofthe object image in the second image based on a feature quantity ofpixels included in the second image; and a highlight section thatperforms a highlight process on the first image based on the type of theobject image in the second image determined by the type determinationsection; wherein, when a plurality of object images are present in thesecond image as the object image: the type determination sectiondetermines one type among first to Nth (N is an integer equal to orlarger than 2) types to which each of the plurality of object images inthe second image belongs, and the highlight section changes a method ofperforming the highlight process among the determined types to which theplurality of object images in the second image belongs.
 2. The imageprocessing device as defined in claim 1, the type determination sectionchanging a feature quantity calculation process corresponding to anorgan to which the object image in the second image belongs.
 3. Theimage processing device as defined in claim 1, the highlight sectionchanging a method of performing the highlight process corresponding toan organ to which the object image in the second image belongs.
 4. Theimage processing device as defined in claim 2, at least one of the firstimage and the second image having been captured by an imaging deviceprovided outside the image processing device, and the highlight sectiondetermining the organ to which the object image in the second imagebelongs based on information about the imaging device that has capturedthe at least one of the first image and the second image.
 5. The imageprocessing device as defined in claim 3, at least one of the first imageand the second image having been captured by an imaging device providedoutside the image processing device, and the highlight sectiondetermining the organ to which the object image in the second imagebelongs based on information about the imaging device that has capturedthe at least one of the first image and the second image.
 6. The imageprocessing device as defined in claim 1, a priority corresponding toeach type being set to each of the first to Nth types, and the highlightsection determining the method of performing the highlight process basedon the priority that is set corresponding to the type of the respectiveobject image in the second image.
 7. The image processing device asdefined in claim 6, the highlight section setting the priority to thetype of the respective object image in the second image corresponding toan organ to which the respective object image in the second imagebelongs.
 8. The image processing device as defined in claim 6, thehighlight section performing the highlight process on one of a pluralityof object images in the first image that belongs to a type with ahighest priority using a highlight method that corresponds to the typeof the object image in the second image.
 9. The image processing deviceas defined in claim 1, the highlight section performing the highlightprocess on one of a plurality of object images in the first image thatbelongs to an ith (1≦i≦N) type among the first to Nth types using ahighlight method that corresponds to the ith type, and performing thehighlight process on the object image in the first image that belongs toa jth j(1≦j≦N, j≠i) type among the first to Nth types using a highlightmethod that corresponds to the jth type.
 10. The image processing deviceas defined in claim 1, the second image being classified into aplurality of types, and the highlight section determining a method ofperforming the highlight process corresponding to the type of therespective object image in the second image and the type of the secondimage.
 11. The image processing device as defined in claim 10, furthercomprising: an area selection section that selects a correspondingattention area from the first image, the corresponding attention areacorresponding to an attention area within the second image that includesthe object image, the highlight section determining the correspondingattention area that is an area corresponding to a blood vessel to be arange subjected to the highlight process when the object image of thesecond image is a blood vessel and the second image is a Narrow BandImaging image.
 12. The image processing device as defined in claim 10,the highlight section determining entirety of the first image to be arange subjected to the highlight process when the object image of thesecond image is a lesion and the second image is a Narrow Band Imagingimage.
 13. The image processing device as defined in claim 1, thehighlight section performing the highlight process that highlights aspecific frequency component of a spatial frequency of the first imageover entirety of the first image.
 14. The image processing device asdefined in claim 1, further comprising: an area selection section thatselects a corresponding attention area from the first image, thecorresponding attention area corresponding to an attention area withinthe second image that includes the object image, the highlight sectionperforming the highlight process that enhances a luminance component ora color component of a signal included in the corresponding attentionarea within the first image based on an edge quantity calculated fromthe second image.
 15. The image processing device as defined in claim 1,further comprising: an area selection section that selects acorresponding attention area from the first image, the correspondingattention area corresponding to an attention area within the secondimage that includes the object image, the highlight section performingthe highlight process that enhances a luminance component or a colorcomponent of a signal included in the corresponding attention areawithin the first image based on a color feature quantity calculated fromthe second image.
 16. The image processing device as defined in claim 1,the feature quantity being a quantity that indicates an edge, hue, orintensity.
 17. The image processing device as defined in claim 1, thetype of the object image in the second image including at least one typeamong a type of lesion and a type of blood vessel.
 18. The imageprocessing device as defined in claim 1, the specific wavelength bandbeing narrower than the wavelength band of the white light.
 19. Theimage processing device as defined in claim 1, each of the first imageand the second image being an in vivo image, and the specific wavelengthband included in each in vivo image being a wavelength band of lightthat is easily absorbed by hemoglobin in blood.
 20. The image processingdevice as defined in claim 19, the specific wavelength band being 390 to445 nm or 530 to 550 nm.
 21. The image processing device as defined inclaim 1, each of the first image and the second image being an in vivoimage, and the specific wavelength band included in each in vivo imagebeing a wavelength band of fluorescence produced by a fluorescentsubstance.
 22. The image processing device as defined in claim 21, thespecific wavelength band being 490 to 625 nm.
 23. The image processingdevice as defined in claim 1, each of the first image and the secondimage being an in vivo image, and the specific wavelength band includedin each in vivo image being a wavelength band of infrared light.
 24. Theimage processing device as defined in claim 23, the specific wavelengthband being 790 to 820 nm or 905 to 970 nm.
 25. The image processingdevice as defined in claim 1, the second image acquisition sectiongenerating the second image based on the first image acquired by thefirst image acquisition section.
 26. The image processing device asdefined in claim 25, the second image acquisition section including asignal extraction section that extracts a signal within the wavelengthband of the white light from the first image acquired by the first imageacquisition section, and the second image acquisition section generatingthe second image that includes a signal within the specific wavelengthband based on the signal extracted by the signal extraction section. 27.The image processing device as defined in claim 26, the second imageacquisition section including a matrix data setting section that setsmatrix data for calculating the signal within the specific wavelengthband from the signal within the wavelength band of the white light, andthe second image acquisition section calculating the signal within thespecific wavelength band from the signal within the wavelength band ofthe white light using the matrix data set by the matrix data settingsection to generate the second image.
 28. An electronic apparatuscomprising the image processing device as defined in claim
 1. 29. Theimage processing device as defined in claim 1, a type of the secondimage including a Narrow Band Imaging image and an Auto FluorescentImaging image.
 30. The image processing device as defined in claim 29,the highlight section changing a method of performing the highlightprocess corresponding to whether the second image is a Narrow BandImaging image or an Auto Fluorescent Imaging image.
 31. An informationstorage device storing a program that causes a computer to function as:a first image acquisition section that acquires a first image thatincludes an object image including information within a wavelength bandof white light; a second image acquisition section that acquires asecond image that includes an object image including information withina specific wavelength band; a type determination section that determinesa type of the object image in the second image based on a featurequantity of pixels included in the second image; and a highlight sectionthat performs a highlight process on the first image based on the typeof the object image in the second image determined by the typedetermination section; wherein, when a plurality of object images arepresent in the second image as the object image: the type determinationsection determines one type among first to Nth (N is an integer equal toor larger than 2) types to which each of the plurality of object imagesin the second image belongs, and the highlight section changes a methodof performing the highlight process among the determined types to whichthe plurality of object images in the second image belongs.
 32. An imageprocessing method comprising: acquiring a first image that includes anobject image including information within a wavelength band of whitelight; acquiring a second image that includes an object image includinginformation within a specific wavelength band; determining a type of theobject image in the second image based on a feature quantity of pixelsincluded in the second image; and performing a highlight process on thefirst image based on the determined type of the object image in thesecond image; wherein, when a plurality of object images are present inthe second image as the object image: the type determination determinesone type among first to Nth (N is an integer equal to or larger than 2)types to which each of the plurality of object images in the secondimage belongs, and the highlight process changes a method of performingthe highlight process among the determined types to which the pluralityof object images in the second image belongs.